{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AutoML with Tabular data - Inference\n",
    "\n",
    "In machine learning, *inference* refers to the process of using a trained predictor to make predictions on data with unknown labels.  First let's produce such a predictor, utilizing bagging/stacking to boost its accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loaded data from: https://autogluon.s3.amazonaws.com/datasets/diabetes/train.csv | Columns = 47 / 47 | Rows = 61059 -> 61059\n",
      "No output_directory specified. Models will be saved in: AutogluonModels/ag-20200801_082821/\n",
      "Beginning AutoGluon training ... Time limit = 120s\n",
      "AutoGluon will save models to AutogluonModels/ag-20200801_082821/\n",
      "AutoGluon Version:  0.0.13b20200731\n",
      "Train Data Rows:    1000\n",
      "Train Data Columns: 47\n",
      "Preprocessing data ...\n",
      "Here are the 3 unique label values in your data:  ['NO', '>30', '<30']\n",
      "AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object).\n",
      "If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "\n",
      "Train Data Class Count: 3\n",
      "Feature Generator processed 1000 data points with 33 features\n",
      "Original Features (raw dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat64 features: 1\n",
      "\tint64 features: 7\n",
      "Original Features (inferred dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "Generated Features (special dtypes):\n",
      "Processed Features (raw dtypes):\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "Processed Features:\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "\tData preprocessing and feature engineering runtime = 0.16s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: accuracy\n",
      "To change this, specify the eval_metric argument of fit()\n",
      "AutoGluon will early stop models using evaluation metric: accuracy\n",
      "Fitting model: RandomForestClassifierGini_STACKER_l0 ... Training model for up to 59.92s of the 119.84s of remaining time.\n",
      "\t0.554\t = Validation accuracy score\n",
      "\t6.9s\t = Training runtime\n",
      "\t1.28s\t = Validation runtime\n",
      "Fitting model: RandomForestClassifierEntr_STACKER_l0 ... Training model for up to 51.27s of the 111.19s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 294 due to low time. Expected time usage reduced from 4.2s -> 4.1s...\n",
      "\t0.553\t = Validation accuracy score\n",
      "\t7.93s\t = Training runtime\n",
      "\t1.28s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierGini_STACKER_l0 ... Training model for up to 41.54s of the 101.46s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 238 due to low time. Expected time usage reduced from 4.2s -> 3.3s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 268 due to low time. Expected time usage reduced from 4.0s -> 3.6s...\n",
      "\t0.509\t = Validation accuracy score\n",
      "\t6.23s\t = Training runtime\n",
      "\t1.3s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierEntr_STACKER_l0 ... Training model for up to 32.83s of the 92.75s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 194 due to low time. Expected time usage reduced from 4.0s -> 2.6s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 210 due to low time. Expected time usage reduced from 4.1s -> 2.9s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 239 due to low time. Expected time usage reduced from 3.9s -> 3.1s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 256 due to low time. Expected time usage reduced from 4.1s -> 3.5s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 296 due to low time. Expected time usage reduced from 4.0s -> 4.0s...\n",
      "\t0.527\t = Validation accuracy score\n",
      "\t6.13s\t = Training runtime\n",
      "\t1.28s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierUnif_STACKER_l0 ... Training model for up to 24.26s of the 84.18s of remaining time.\n",
      "\t0.44\t = Validation accuracy score\n",
      "\t0.11s\t = Training runtime\n",
      "\t1.07s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierDist_STACKER_l0 ... Training model for up to 23.05s of the 82.97s of remaining time.\n",
      "\t0.467\t = Validation accuracy score\n",
      "\t0.11s\t = Training runtime\n",
      "\t1.07s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifier_STACKER_l0 ... Training model for up to 21.84s of the 81.76s of remaining time.\n",
      "\t0.585\t = Validation accuracy score\n",
      "\t7.05s\t = Training runtime\n",
      "\t0.31s\t = Validation runtime\n",
      "Fitting model: CatboostClassifier_STACKER_l0 ... Training model for up to 14.4s of the 74.32s of remaining time.\n",
      "\t0.582\t = Validation accuracy score\n",
      "\t13.74s\t = Training runtime\n",
      "\t0.39s\t = Validation runtime\n",
      "Fitting model: NeuralNetClassifier_STACKER_l0 ... Training model for up to 0.21s of the 60.13s of remaining time.\n",
      "\tRan out of time, stopping training early.\n",
      "\tTime limit exceeded... Skipping NeuralNetClassifier_STACKER_l0.\n",
      "Completed 1/20 k-fold bagging repeats ...\n",
      "Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 119.84s of the 59.31s of remaining time.\n",
      "\t0.585\t = Validation accuracy score\n",
      "\t0.74s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Fitting model: RandomForestClassifierGini_STACKER_l1 ... Training model for up to 58.54s of the 58.53s of remaining time.\n",
      "\t0.553\t = Validation accuracy score\n",
      "\t10.36s\t = Training runtime\n",
      "\t1.39s\t = Validation runtime\n",
      "Fitting model: RandomForestClassifierEntr_STACKER_l1 ... Training model for up to 46.29s of the 46.28s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 269 due to low time. Expected time usage reduced from 4.1s -> 3.7s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 284 due to low time. Expected time usage reduced from 4.2s -> 4.0s...\n",
      "\t0.555\t = Validation accuracy score\n",
      "\t10.11s\t = Training runtime\n",
      "\t1.34s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierGini_STACKER_l1 ... Training model for up to 34.41s of the 34.4s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 192 due to low time. Expected time usage reduced from 4.3s -> 2.8s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 204 due to low time. Expected time usage reduced from 4.4s -> 3.0s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 241 due to low time. Expected time usage reduced from 4.1s -> 3.3s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 270 due to low time. Expected time usage reduced from 4.1s -> 3.7s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 297 due to low time. Expected time usage reduced from 4.2s -> 4.2s...\n",
      "\t0.543\t = Validation accuracy score\n",
      "\t6.25s\t = Training runtime\n",
      "\t1.26s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierEntr_STACKER_l1 ... Training model for up to 26.14s of the 26.13s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 152 due to low time. Expected time usage reduced from 4.1s -> 2.1s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 171 due to low time. Expected time usage reduced from 4.0s -> 2.3s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 181 due to low time. Expected time usage reduced from 4.1s -> 2.5s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 202 due to low time. Expected time usage reduced from 4.1s -> 2.8s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 239 due to low time. Expected time usage reduced from 3.9s -> 3.2s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 271 due to low time. Expected time usage reduced from 4.1s -> 3.7s...\n",
      "\t0.538\t = Validation accuracy score\n",
      "\t5.99s\t = Training runtime\n",
      "\t1.27s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierUnif_STACKER_l1 ... Training model for up to 18.11s of the 18.1s of remaining time.\n",
      "\t0.452\t = Validation accuracy score\n",
      "\t0.16s\t = Training runtime\n",
      "\t1.05s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierDist_STACKER_l1 ... Training model for up to 16.84s of the 16.83s of remaining time.\n",
      "\t0.473\t = Validation accuracy score\n",
      "\t0.13s\t = Training runtime\n",
      "\t1.07s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifier_STACKER_l1 ... Training model for up to 15.58s of the 15.57s of remaining time.\n",
      "\tRan out of time, early stopping on iteration 54. Best iteration is:\n",
      "\t[3]\ttrain_set's multi_error: 0.306667\tvalid_set's multi_error: 0.42\n",
      "\tRan out of time, early stopping on iteration 58. Best iteration is:\n",
      "\t[16]\ttrain_set's multi_error: 0.105556\tvalid_set's multi_error: 0.44\n",
      "\tRan out of time, early stopping on iteration 39. Best iteration is:\n",
      "\t[8]\ttrain_set's multi_error: 0.192222\tvalid_set's multi_error: 0.37\n",
      "\tRan out of time, early stopping on iteration 58. Best iteration is:\n",
      "\t[2]\ttrain_set's multi_error: 0.358889\tvalid_set's multi_error: 0.43\n",
      "\tRan out of time, early stopping on iteration 88. Best iteration is:\n",
      "\t[21]\ttrain_set's multi_error: 0.0677778\tvalid_set's multi_error: 0.46\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 89. Best iteration is:\n",
      "\t[7]\ttrain_set's multi_error: 0.215556\tvalid_set's multi_error: 0.39\n",
      "\tRan out of time, early stopping on iteration 83. Best iteration is:\n",
      "\t[16]\ttrain_set's multi_error: 0.114444\tvalid_set's multi_error: 0.39\n",
      "\tRan out of time, early stopping on iteration 85. Best iteration is:\n",
      "\t[22]\ttrain_set's multi_error: 0.0466667\tvalid_set's multi_error: 0.45\n",
      "\tRan out of time, early stopping on iteration 106. Best iteration is:\n",
      "\t[3]\ttrain_set's multi_error: 0.287778\tvalid_set's multi_error: 0.45\n",
      "\tRan out of time, early stopping on iteration 111. Best iteration is:\n",
      "\t[11]\ttrain_set's multi_error: 0.17\tvalid_set's multi_error: 0.46\n",
      "\t0.574\t = Validation accuracy score\n",
      "\t14.71s\t = Training runtime\n",
      "\t0.41s\t = Validation runtime\n",
      "Fitting model: CatboostClassifier_STACKER_l1 ... Training model for up to 0.32s of the 0.31s of remaining time.\n",
      "\tTime limit exceeded... Skipping CatboostClassifier_STACKER_l1.\n",
      "Fitting model: NeuralNetClassifier_STACKER_l1 ... Training model for up to 0.18s of the 0.17s of remaining time.\n",
      "\tRan out of time, stopping training early.\n",
      "\tTime limit exceeded... Skipping NeuralNetClassifier_STACKER_l1.\n",
      "Completed 1/20 k-fold bagging repeats ...\n",
      "Fitting model: weighted_ensemble_k0_l2 ... Training model for up to 119.84s of the -0.43s of remaining time.\n",
      "\t0.574\t = Validation accuracy score\n",
      "\t0.39s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 120.86s ...\n",
      "Loaded data from: https://autogluon.s3.amazonaws.com/datasets/diabetes/test.csv | Columns = 47 / 47 | Rows = 20354 -> 20354\n",
      "Evaluation: accuracy on test data: 0.53\n",
      "Evaluations on test data:\n",
      "{\n",
      "    \"accuracy\": 0.53,\n",
      "    \"accuracy_score\": 0.53,\n",
      "    \"balanced_accuracy_score\": 0.3654103642335613,\n",
      "    \"matthews_corrcoef\": 0.08145046100670611\n",
      "}\n",
      "/Users/jonasmue/virtual/agmaster/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "Detailed (per-class) classification report:\n",
      "{\n",
      "    \"<30\": {\n",
      "        \"precision\": 0.0,\n",
      "        \"recall\": 0.0,\n",
      "        \"f1-score\": 0.0,\n",
      "        \"support\": 97\n",
      "    },\n",
      "    \">30\": {\n",
      "        \"precision\": 0.40181268882175225,\n",
      "        \"recall\": 0.39233038348082594,\n",
      "        \"f1-score\": 0.3970149253731343,\n",
      "        \"support\": 339\n",
      "    },\n",
      "    \"NO\": {\n",
      "        \"precision\": 0.593423019431988,\n",
      "        \"recall\": 0.7039007092198581,\n",
      "        \"f1-score\": 0.6439578264395783,\n",
      "        \"support\": 564\n",
      "    },\n",
      "    \"accuracy\": 0.53,\n",
      "    \"macro avg\": {\n",
      "        \"precision\": 0.3317452360845801,\n",
      "        \"recall\": 0.3654103642335613,\n",
      "        \"f1-score\": 0.3469909172709042,\n",
      "        \"support\": 1000\n",
      "    },\n",
      "    \"weighted avg\": {\n",
      "        \"precision\": 0.47090508447021523,\n",
      "        \"recall\": 0.53,\n",
      "        \"f1-score\": 0.4977802738134147,\n",
      "        \"support\": 1000\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "from autogluon import TabularPrediction as task\n",
    "from IPython.display import display\n",
    "import numpy as np\n",
    "import pprint\n",
    "\n",
    "subsample_size = 1000 # experiment with larger values to try AutoGluon with larger datasets \n",
    "time_limits = 120 # experiment with larger values to get a better sense of the achievable accuracy\n",
    "\n",
    "train_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/diabetes/train.csv')\n",
    "train_data = train_data.head(subsample_size) # subsample data for faster demo\n",
    "label_column = 'readmitted'\n",
    "\n",
    "predictor = task.fit(train_data=train_data, label=label_column, auto_stack=True, time_limits=time_limits)\n",
    "\n",
    "test_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/diabetes/test.csv')\n",
    "test_data = test_data.head(subsample_size) # subsample data for faster demo\n",
    "y_test = test_data[label_column] # ground-truth target values\n",
    "test_data_nolab = test_data.drop(labels=[label_column],axis=1) # delete label column to prove we're not cheating\n",
    "\n",
    "y_pred = predictor.predict(test_data_nolab)\n",
    "perf = predictor.evaluate_predictions(y_true=y_test, y_pred=y_pred, auxiliary_metrics=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the above predictions are produced by the model AutoGluon believes to be the most accurate, recall we can also evaluate the predictive performance of every model AutoGluon has trained:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                    model  score_test  score_val  pred_time_test  pred_time_val   fit_time  pred_time_test_marginal  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer\n",
      "0           LightGBMClassifier_STACKER_l1       0.551      0.574        9.478982       8.360810  62.905900                 0.498886                0.410340          14.708010            1       True\n",
      "1                 weighted_ensemble_k0_l2       0.551      0.574        9.481465       8.363896  63.292353                 0.002483                0.003086           0.386453            2       True\n",
      "2           CatboostClassifier_STACKER_l0       0.550      0.582        0.113835       0.385099  13.741464                 0.113835                0.385099          13.741464            0       True\n",
      "3   RandomForestClassifierGini_STACKER_l1       0.542      0.553       10.614846       9.339597  58.561704                 1.634750                1.389127          10.363814            1       True\n",
      "4   RandomForestClassifierGini_STACKER_l0       0.541      0.554        1.283559       1.277415   6.900042                 1.283559                1.277415           6.900042            0       True\n",
      "5     ExtraTreesClassifierEntr_STACKER_l0       0.540      0.527        1.700516       1.275319   6.131904                 1.700516                1.275319           6.131904            0       True\n",
      "6     ExtraTreesClassifierGini_STACKER_l0       0.538      0.509        1.646748       1.295418   6.231692                 1.646748                1.295418           6.231692            0       True\n",
      "7   RandomForestClassifierEntr_STACKER_l0       0.538      0.553        1.682435       1.279882   7.926347                 1.682435                1.279882           7.926347            0       True\n",
      "8   RandomForestClassifierEntr_STACKER_l1       0.538      0.555       10.943728       9.287851  58.306758                 1.963632                1.337381          10.108868            1       True\n",
      "9     ExtraTreesClassifierEntr_STACKER_l1       0.535      0.538       10.471045       9.220299  54.190841                 1.490949                1.269828           5.992951            1       True\n",
      "10    ExtraTreesClassifierGini_STACKER_l1       0.535      0.543       10.966115       9.212859  54.446588                 1.986019                1.262388           6.248698            1       True\n",
      "11          LightGBMClassifier_STACKER_l0       0.530      0.585        0.431814       0.306638   7.051510                 0.431814                0.306638           7.051510            0       True\n",
      "12                weighted_ensemble_k0_l1       0.530      0.585        0.434921       0.311344   7.796096                 0.003107                0.004706           0.744586            1       True\n",
      "13    KNeighborsClassifierDist_STACKER_l1       0.477      0.473       10.187279       9.021663  48.332786                 1.207184                1.071193           0.134896            1       True\n",
      "14    KNeighborsClassifierUnif_STACKER_l0       0.468      0.440        1.068679       1.065157   0.109639                 1.068679                1.065157           0.109639            0       True\n",
      "15    KNeighborsClassifierUnif_STACKER_l1       0.468      0.452       10.085832       9.001932  48.356426                 1.105736                1.051461           0.158536            1       True\n",
      "16    KNeighborsClassifierDist_STACKER_l0       0.467      0.467        1.052510       1.065543   0.105294                 1.052510                1.065543           0.105294            0       True\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>score_test</th>\n",
       "      <th>score_val</th>\n",
       "      <th>pred_time_test</th>\n",
       "      <th>pred_time_val</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>pred_time_test_marginal</th>\n",
       "      <th>pred_time_val_marginal</th>\n",
       "      <th>fit_time_marginal</th>\n",
       "      <th>stack_level</th>\n",
       "      <th>can_infer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LightGBMClassifier_STACKER_l1</td>\n",
       "      <td>0.551</td>\n",
       "      <td>0.574</td>\n",
       "      <td>9.478982</td>\n",
       "      <td>8.360810</td>\n",
       "      <td>62.905900</td>\n",
       "      <td>0.498886</td>\n",
       "      <td>0.410340</td>\n",
       "      <td>14.708010</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>weighted_ensemble_k0_l2</td>\n",
       "      <td>0.551</td>\n",
       "      <td>0.574</td>\n",
       "      <td>9.481465</td>\n",
       "      <td>8.363896</td>\n",
       "      <td>63.292353</td>\n",
       "      <td>0.002483</td>\n",
       "      <td>0.003086</td>\n",
       "      <td>0.386453</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CatboostClassifier_STACKER_l0</td>\n",
       "      <td>0.550</td>\n",
       "      <td>0.582</td>\n",
       "      <td>0.113835</td>\n",
       "      <td>0.385099</td>\n",
       "      <td>13.741464</td>\n",
       "      <td>0.113835</td>\n",
       "      <td>0.385099</td>\n",
       "      <td>13.741464</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RandomForestClassifierGini_STACKER_l1</td>\n",
       "      <td>0.542</td>\n",
       "      <td>0.553</td>\n",
       "      <td>10.614846</td>\n",
       "      <td>9.339597</td>\n",
       "      <td>58.561704</td>\n",
       "      <td>1.634750</td>\n",
       "      <td>1.389127</td>\n",
       "      <td>10.363814</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RandomForestClassifierGini_STACKER_l0</td>\n",
       "      <td>0.541</td>\n",
       "      <td>0.554</td>\n",
       "      <td>1.283559</td>\n",
       "      <td>1.277415</td>\n",
       "      <td>6.900042</td>\n",
       "      <td>1.283559</td>\n",
       "      <td>1.277415</td>\n",
       "      <td>6.900042</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ExtraTreesClassifierEntr_STACKER_l0</td>\n",
       "      <td>0.540</td>\n",
       "      <td>0.527</td>\n",
       "      <td>1.700516</td>\n",
       "      <td>1.275319</td>\n",
       "      <td>6.131904</td>\n",
       "      <td>1.700516</td>\n",
       "      <td>1.275319</td>\n",
       "      <td>6.131904</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ExtraTreesClassifierGini_STACKER_l0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>0.509</td>\n",
       "      <td>1.646748</td>\n",
       "      <td>1.295418</td>\n",
       "      <td>6.231692</td>\n",
       "      <td>1.646748</td>\n",
       "      <td>1.295418</td>\n",
       "      <td>6.231692</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>RandomForestClassifierEntr_STACKER_l0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>0.553</td>\n",
       "      <td>1.682435</td>\n",
       "      <td>1.279882</td>\n",
       "      <td>7.926347</td>\n",
       "      <td>1.682435</td>\n",
       "      <td>1.279882</td>\n",
       "      <td>7.926347</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>RandomForestClassifierEntr_STACKER_l1</td>\n",
       "      <td>0.538</td>\n",
       "      <td>0.555</td>\n",
       "      <td>10.943728</td>\n",
       "      <td>9.287851</td>\n",
       "      <td>58.306758</td>\n",
       "      <td>1.963632</td>\n",
       "      <td>1.337381</td>\n",
       "      <td>10.108868</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>ExtraTreesClassifierEntr_STACKER_l1</td>\n",
       "      <td>0.535</td>\n",
       "      <td>0.538</td>\n",
       "      <td>10.471045</td>\n",
       "      <td>9.220299</td>\n",
       "      <td>54.190841</td>\n",
       "      <td>1.490949</td>\n",
       "      <td>1.269828</td>\n",
       "      <td>5.992951</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>ExtraTreesClassifierGini_STACKER_l1</td>\n",
       "      <td>0.535</td>\n",
       "      <td>0.543</td>\n",
       "      <td>10.966115</td>\n",
       "      <td>9.212859</td>\n",
       "      <td>54.446588</td>\n",
       "      <td>1.986019</td>\n",
       "      <td>1.262388</td>\n",
       "      <td>6.248698</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>LightGBMClassifier_STACKER_l0</td>\n",
       "      <td>0.530</td>\n",
       "      <td>0.585</td>\n",
       "      <td>0.431814</td>\n",
       "      <td>0.306638</td>\n",
       "      <td>7.051510</td>\n",
       "      <td>0.431814</td>\n",
       "      <td>0.306638</td>\n",
       "      <td>7.051510</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>weighted_ensemble_k0_l1</td>\n",
       "      <td>0.530</td>\n",
       "      <td>0.585</td>\n",
       "      <td>0.434921</td>\n",
       "      <td>0.311344</td>\n",
       "      <td>7.796096</td>\n",
       "      <td>0.003107</td>\n",
       "      <td>0.004706</td>\n",
       "      <td>0.744586</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>KNeighborsClassifierDist_STACKER_l1</td>\n",
       "      <td>0.477</td>\n",
       "      <td>0.473</td>\n",
       "      <td>10.187279</td>\n",
       "      <td>9.021663</td>\n",
       "      <td>48.332786</td>\n",
       "      <td>1.207184</td>\n",
       "      <td>1.071193</td>\n",
       "      <td>0.134896</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>KNeighborsClassifierUnif_STACKER_l0</td>\n",
       "      <td>0.468</td>\n",
       "      <td>0.440</td>\n",
       "      <td>1.068679</td>\n",
       "      <td>1.065157</td>\n",
       "      <td>0.109639</td>\n",
       "      <td>1.068679</td>\n",
       "      <td>1.065157</td>\n",
       "      <td>0.109639</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>KNeighborsClassifierUnif_STACKER_l1</td>\n",
       "      <td>0.468</td>\n",
       "      <td>0.452</td>\n",
       "      <td>10.085832</td>\n",
       "      <td>9.001932</td>\n",
       "      <td>48.356426</td>\n",
       "      <td>1.105736</td>\n",
       "      <td>1.051461</td>\n",
       "      <td>0.158536</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>KNeighborsClassifierDist_STACKER_l0</td>\n",
       "      <td>0.467</td>\n",
       "      <td>0.467</td>\n",
       "      <td>1.052510</td>\n",
       "      <td>1.065543</td>\n",
       "      <td>0.105294</td>\n",
       "      <td>1.052510</td>\n",
       "      <td>1.065543</td>\n",
       "      <td>0.105294</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    model  score_test  score_val  \\\n",
       "0           LightGBMClassifier_STACKER_l1       0.551      0.574   \n",
       "1                 weighted_ensemble_k0_l2       0.551      0.574   \n",
       "2           CatboostClassifier_STACKER_l0       0.550      0.582   \n",
       "3   RandomForestClassifierGini_STACKER_l1       0.542      0.553   \n",
       "4   RandomForestClassifierGini_STACKER_l0       0.541      0.554   \n",
       "5     ExtraTreesClassifierEntr_STACKER_l0       0.540      0.527   \n",
       "6     ExtraTreesClassifierGini_STACKER_l0       0.538      0.509   \n",
       "7   RandomForestClassifierEntr_STACKER_l0       0.538      0.553   \n",
       "8   RandomForestClassifierEntr_STACKER_l1       0.538      0.555   \n",
       "9     ExtraTreesClassifierEntr_STACKER_l1       0.535      0.538   \n",
       "10    ExtraTreesClassifierGini_STACKER_l1       0.535      0.543   \n",
       "11          LightGBMClassifier_STACKER_l0       0.530      0.585   \n",
       "12                weighted_ensemble_k0_l1       0.530      0.585   \n",
       "13    KNeighborsClassifierDist_STACKER_l1       0.477      0.473   \n",
       "14    KNeighborsClassifierUnif_STACKER_l0       0.468      0.440   \n",
       "15    KNeighborsClassifierUnif_STACKER_l1       0.468      0.452   \n",
       "16    KNeighborsClassifierDist_STACKER_l0       0.467      0.467   \n",
       "\n",
       "    pred_time_test  pred_time_val   fit_time  pred_time_test_marginal  \\\n",
       "0         9.478982       8.360810  62.905900                 0.498886   \n",
       "1         9.481465       8.363896  63.292353                 0.002483   \n",
       "2         0.113835       0.385099  13.741464                 0.113835   \n",
       "3        10.614846       9.339597  58.561704                 1.634750   \n",
       "4         1.283559       1.277415   6.900042                 1.283559   \n",
       "5         1.700516       1.275319   6.131904                 1.700516   \n",
       "6         1.646748       1.295418   6.231692                 1.646748   \n",
       "7         1.682435       1.279882   7.926347                 1.682435   \n",
       "8        10.943728       9.287851  58.306758                 1.963632   \n",
       "9        10.471045       9.220299  54.190841                 1.490949   \n",
       "10       10.966115       9.212859  54.446588                 1.986019   \n",
       "11        0.431814       0.306638   7.051510                 0.431814   \n",
       "12        0.434921       0.311344   7.796096                 0.003107   \n",
       "13       10.187279       9.021663  48.332786                 1.207184   \n",
       "14        1.068679       1.065157   0.109639                 1.068679   \n",
       "15       10.085832       9.001932  48.356426                 1.105736   \n",
       "16        1.052510       1.065543   0.105294                 1.052510   \n",
       "\n",
       "    pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  \n",
       "0                 0.410340          14.708010            1       True  \n",
       "1                 0.003086           0.386453            2       True  \n",
       "2                 0.385099          13.741464            0       True  \n",
       "3                 1.389127          10.363814            1       True  \n",
       "4                 1.277415           6.900042            0       True  \n",
       "5                 1.275319           6.131904            0       True  \n",
       "6                 1.295418           6.231692            0       True  \n",
       "7                 1.279882           7.926347            0       True  \n",
       "8                 1.337381          10.108868            1       True  \n",
       "9                 1.269828           5.992951            1       True  \n",
       "10                1.262388           6.248698            1       True  \n",
       "11                0.306638           7.051510            0       True  \n",
       "12                0.004706           0.744586            1       True  \n",
       "13                1.071193           0.134896            1       True  \n",
       "14                1.065157           0.109639            0       True  \n",
       "15                1.051461           0.158536            1       True  \n",
       "16                1.065543           0.105294            0       True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_perf = predictor.leaderboard(test_data)\n",
    "display(test_perf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that different models not only have different predictive accuracy, but also take differing amounts of time to compute predictions (`pred_time_test`). Low latency inference may be important for ML deployed in settings where decisions must be made in real-time, one datapoint at a time.  While the model ensembles are typically the most accurate, they are also generally higher latency with larger memory footprint than individual models.\n",
    "\n",
    "When we call `predict()`, AutoGluon automatically predicts with the model that displayed the best validation performance (this is typically the weighted-ensemble at the topmost stack-layer). Recall we can instead specify predictions should be made with a certain model as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['NO', 'NO', 'NO', 'NO', 'NO', '>30', '>30', '>30', 'NO', 'NO',\n",
       "       'NO', 'NO', '>30', 'NO', 'NO', 'NO', '>30', 'NO', '>30', 'NO'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.predict(test_data[:20], model='LightGBMClassifier_STACKER_l0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'LightGBMClassifier_STACKER_l0',\n",
       " 'model_type': 'StackerEnsembleModel',\n",
       " 'problem_type': 'multiclass',\n",
       " 'eval_metric': 'accuracy',\n",
       " 'stopping_metric': 'accuracy',\n",
       " 'fit_time': 7.051509618759155,\n",
       " 'predict_time': 0.3066377639770508,\n",
       " 'val_score': 0.585,\n",
       " 'hyperparameters': {'max_models': 25, 'max_models_per_type': 5},\n",
       " 'hyperparameters_fit': {},\n",
       " 'hyperparameters_nondefault': [],\n",
       " 'memory_size': 2744,\n",
       " 'bagged_info': {'child_type': 'LGBModel',\n",
       "  'num_child_models': 10,\n",
       "  'child_model_names': ['LightGBMClassifier_fold_0',\n",
       "   'LightGBMClassifier_fold_1',\n",
       "   'LightGBMClassifier_fold_2',\n",
       "   'LightGBMClassifier_fold_3',\n",
       "   'LightGBMClassifier_fold_4',\n",
       "   'LightGBMClassifier_fold_5',\n",
       "   'LightGBMClassifier_fold_6',\n",
       "   'LightGBMClassifier_fold_7',\n",
       "   'LightGBMClassifier_fold_8',\n",
       "   'LightGBMClassifier_fold_9'],\n",
       "  '_n_repeats': 1,\n",
       "  '_k_per_n_repeat': [10],\n",
       "  '_random_state': 0,\n",
       "  'low_memory': True,\n",
       "  'bagged_mode': True,\n",
       "  'max_memory_size': 2654226,\n",
       "  'min_memory_size': 727045},\n",
       " 'stacker_info': {'num_base_models': 0,\n",
       "  'base_model_names': [],\n",
       "  'use_orig_features': True},\n",
       " 'children_info': {'LightGBMClassifier_fold_0': {'name': 'LightGBMClassifier_fold_0',\n",
       "   'model_type': 'LGBModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 1.1528139114379883,\n",
       "   'predict_time': 0.03233218193054199,\n",
       "   'val_score': 0.59,\n",
       "   'hyperparameters': {'num_boost_round': 10000,\n",
       "    'num_threads': -1,\n",
       "    'objective': 'multiclass',\n",
       "    'num_classes': 3,\n",
       "    'verbose': -1,\n",
       "    'boosting_type': 'gbdt',\n",
       "    'two_round': True},\n",
       "   'hyperparameters_fit': {'num_boost_round': 21},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 218206},\n",
       "  'LightGBMClassifier_fold_1': {'name': 'LightGBMClassifier_fold_1',\n",
       "   'model_type': 'LGBModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.848334789276123,\n",
       "   'predict_time': 0.025756359100341797,\n",
       "   'val_score': 0.57,\n",
       "   'hyperparameters': {'num_boost_round': 10000,\n",
       "    'num_threads': -1,\n",
       "    'objective': 'multiclass',\n",
       "    'num_classes': 3,\n",
       "    'verbose': -1,\n",
       "    'boosting_type': 'gbdt',\n",
       "    'two_round': True},\n",
       "   'hyperparameters_fit': {'num_boost_round': 72},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 724301},\n",
       "  'LightGBMClassifier_fold_2': {'name': 'LightGBMClassifier_fold_2',\n",
       "   'model_type': 'LGBModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.47027111053466797,\n",
       "   'predict_time': 0.0259859561920166,\n",
       "   'val_score': 0.58,\n",
       "   'hyperparameters': {'num_boost_round': 10000,\n",
       "    'num_threads': -1,\n",
       "    'objective': 'multiclass',\n",
       "    'num_classes': 3,\n",
       "    'verbose': -1,\n",
       "    'boosting_type': 'gbdt',\n",
       "    'two_round': True},\n",
       "   'hyperparameters_fit': {'num_boost_round': 15},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 159144},\n",
       "  'LightGBMClassifier_fold_3': {'name': 'LightGBMClassifier_fold_3',\n",
       "   'model_type': 'LGBModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.5611393451690674,\n",
       "   'predict_time': 0.025284767150878906,\n",
       "   'val_score': 0.6,\n",
       "   'hyperparameters': {'num_boost_round': 10000,\n",
       "    'num_threads': -1,\n",
       "    'objective': 'multiclass',\n",
       "    'num_classes': 3,\n",
       "    'verbose': -1,\n",
       "    'boosting_type': 'gbdt',\n",
       "    'two_round': True},\n",
       "   'hyperparameters_fit': {'num_boost_round': 14},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 148881},\n",
       "  'LightGBMClassifier_fold_4': {'name': 'LightGBMClassifier_fold_4',\n",
       "   'model_type': 'LGBModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.6059088706970215,\n",
       "   'predict_time': 0.028323888778686523,\n",
       "   'val_score': 0.55,\n",
       "   'hyperparameters': {'num_boost_round': 10000,\n",
       "    'num_threads': -1,\n",
       "    'objective': 'multiclass',\n",
       "    'num_classes': 3,\n",
       "    'verbose': -1,\n",
       "    'boosting_type': 'gbdt',\n",
       "    'two_round': True},\n",
       "   'hyperparameters_fit': {'num_boost_round': 15},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 158279},\n",
       "  'LightGBMClassifier_fold_5': {'name': 'LightGBMClassifier_fold_5',\n",
       "   'model_type': 'LGBModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 1.4242899417877197,\n",
       "   'predict_time': 0.03960824012756348,\n",
       "   'val_score': 0.67,\n",
       "   'hyperparameters': {'num_boost_round': 10000,\n",
       "    'num_threads': -1,\n",
       "    'objective': 'multiclass',\n",
       "    'num_classes': 3,\n",
       "    'verbose': -1,\n",
       "    'boosting_type': 'gbdt',\n",
       "    'two_round': True},\n",
       "   'hyperparameters_fit': {'num_boost_round': 71},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 714648},\n",
       "  'LightGBMClassifier_fold_6': {'name': 'LightGBMClassifier_fold_6',\n",
       "   'model_type': 'LGBModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.4203641414642334,\n",
       "   'predict_time': 0.031091928482055664,\n",
       "   'val_score': 0.57,\n",
       "   'hyperparameters': {'num_boost_round': 10000,\n",
       "    'num_threads': -1,\n",
       "    'objective': 'multiclass',\n",
       "    'num_classes': 3,\n",
       "    'verbose': -1,\n",
       "    'boosting_type': 'gbdt',\n",
       "    'two_round': True},\n",
       "   'hyperparameters_fit': {'num_boost_round': 7},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 80170},\n",
       "  'LightGBMClassifier_fold_7': {'name': 'LightGBMClassifier_fold_7',\n",
       "   'model_type': 'LGBModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.4742739200592041,\n",
       "   'predict_time': 0.03865313529968262,\n",
       "   'val_score': 0.61,\n",
       "   'hyperparameters': {'num_boost_round': 10000,\n",
       "    'num_threads': -1,\n",
       "    'objective': 'multiclass',\n",
       "    'num_classes': 3,\n",
       "    'verbose': -1,\n",
       "    'boosting_type': 'gbdt',\n",
       "    'two_round': True},\n",
       "   'hyperparameters_fit': {'num_boost_round': 7},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 79837},\n",
       "  'LightGBMClassifier_fold_8': {'name': 'LightGBMClassifier_fold_8',\n",
       "   'model_type': 'LGBModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.5879428386688232,\n",
       "   'predict_time': 0.030839204788208008,\n",
       "   'val_score': 0.58,\n",
       "   'hyperparameters': {'num_boost_round': 10000,\n",
       "    'num_threads': -1,\n",
       "    'objective': 'multiclass',\n",
       "    'num_classes': 3,\n",
       "    'verbose': -1,\n",
       "    'boosting_type': 'gbdt',\n",
       "    'two_round': True},\n",
       "   'hyperparameters_fit': {'num_boost_round': 33},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 337426},\n",
       "  'LightGBMClassifier_fold_9': {'name': 'LightGBMClassifier_fold_9',\n",
       "   'model_type': 'LGBModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.5061707496643066,\n",
       "   'predict_time': 0.028762102127075195,\n",
       "   'val_score': 0.53,\n",
       "   'hyperparameters': {'num_boost_round': 10000,\n",
       "    'num_threads': -1,\n",
       "    'objective': 'multiclass',\n",
       "    'num_classes': 3,\n",
       "    'verbose': -1,\n",
       "    'boosting_type': 'gbdt',\n",
       "    'two_round': True},\n",
       "   'hyperparameters_fit': {'num_boost_round': 2},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 30590}}}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "gbm_stacker = predictor._trainer.load_model('LightGBMClassifier_STACKER_l0')\n",
    "display(gbm_stacker.get_info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the **LightGBMClassifier_STACKER_l0** model is a bagged ensemble (since we are using stacking), which recall actually involves 10 different LightGBM models trained on different train/validation folds. We can collapse this bag of 10 models into a single **LightGBMClassifier** model that's fit to the full dataset. Since this model has no validation data to gauge performance after each boosting round, it will early-stop based on the average of the early-stopping points previously used by the 10 LightGBM models in the bag (which were each trained with validation-based early-stopping)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: LightGBMClassifier_FULL_STACKER_l0 ...\n",
      "\t0.35s\t = Training runtime\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name of refit-full model corresponding to the previous bagged ensemble:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'LightGBMClassifier_STACKER_l0': 'LightGBMClassifier_FULL_STACKER_l0'}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "refit_model_map = predictor.refit_full(model='LightGBMClassifier_STACKER_l0')\n",
    "print(\"Name of refit-full model corresponding to the previous bagged ensemble:\")\n",
    "display(refit_model_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['NO' 'NO' 'NO' 'NO' 'NO' '>30' '>30' '>30' 'NO' 'NO' 'NO' 'NO' '>30' 'NO'\n",
      " 'NO' 'NO' '>30' 'NO' '>30' 'NO']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'name': 'LightGBMClassifier_FULL_STACKER_l0',\n",
       " 'model_type': 'StackerEnsembleModel',\n",
       " 'problem_type': 'multiclass',\n",
       " 'eval_metric': 'accuracy',\n",
       " 'stopping_metric': 'accuracy',\n",
       " 'fit_time': 0.3497910499572754,\n",
       " 'predict_time': None,\n",
       " 'val_score': None,\n",
       " 'hyperparameters': {'max_models': 25, 'max_models_per_type': 5},\n",
       " 'hyperparameters_fit': {},\n",
       " 'hyperparameters_nondefault': [],\n",
       " 'memory_size': 2393,\n",
       " 'bagged_info': {'child_type': 'LGBModel',\n",
       "  'num_child_models': 1,\n",
       "  'child_model_names': ['LightGBMClassifier_FULL'],\n",
       "  '_n_repeats': 1,\n",
       "  '_k_per_n_repeat': [1],\n",
       "  '_random_state': 0,\n",
       "  'low_memory': True,\n",
       "  'bagged_mode': False,\n",
       "  'max_memory_size': 271369,\n",
       "  'min_memory_size': 271369},\n",
       " 'stacker_info': {'num_base_models': 0,\n",
       "  'base_model_names': [],\n",
       "  'use_orig_features': True},\n",
       " 'children_info': {'LightGBMClassifier_FULL': {'name': 'LightGBMClassifier_FULL',\n",
       "   'model_type': 'LGBModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.3497910499572754,\n",
       "   'predict_time': None,\n",
       "   'val_score': None,\n",
       "   'hyperparameters': {'num_boost_round': 26,\n",
       "    'num_threads': -1,\n",
       "    'objective': 'multiclass',\n",
       "    'num_classes': 3,\n",
       "    'verbose': -1,\n",
       "    'boosting_type': 'gbdt',\n",
       "    'two_round': True},\n",
       "   'hyperparameters_fit': {'num_boost_round': 0},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 268976}}}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "single_gbm_preds = predictor.predict(test_data[:20], model='LightGBMClassifier_FULL_STACKER_l0')\n",
    "print(single_gbm_preds)\n",
    "\n",
    "single_gbm = predictor._trainer.load_model('LightGBMClassifier_FULL_STACKER_l0')\n",
    "display(single_gbm.get_info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This single GBM model requires 10x less memory than its bagged counterpart and can produce predictions 10x faster as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Distillation \n",
    "\n",
    "While computationally-favorable, single individual models will usually have lower accuracy than weighted/stacked/bagged ensembles. *Model Distillation* offers one way the retain the computational benefits of a single model, while enjoying some of the accuracy-boost that comes with ensembling. The idea is to train the individual model (which we can call the *student*) to mimic the predictions of the full stack ensemble (the *teacher*). Rather than fitting the individual model to the training data target-values, one can fit this student model to an alternative dataset whose target values are the predictions from the ensemble teacher ([Bucila et al, 2006](https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf)). \n",
    "In classification tasks, the teacher provides its predicted class-probabilities as targets for the student model, which may encode richer information about class-similarities or label-noise, and we can also blend the teacher predictions with the original target-values from the training data for further improvement  ([Hinton et al, 2014](https://arxiv.org/abs/1503.02531)). \n",
    "\n",
    "Distillation in AutoGluon follows the strategy proposed in \n",
    "([Fakoor et al, 2020](https://arxiv.org/abs/2006.14284)) which we call *augmented distillation* and depict below.\n",
    "\n",
    "<img src=\"files/images/distillfigure.png\" width=\"700\" height=\"400\">\n",
    "\n",
    "The overall augmented distillation procedure involves 4 steps:\n",
    "\n",
    "1. **Fit** stack ensemble to the original training data via: `fit(train_data, label_column, auto_stack=True)`\n",
    "\n",
    "2. Use the training data features to **generate** a synthetic dataset of features (we call this the *augmented data*). The augmented data should ideally follow a similar underlying feature distribution as the training data, and can either be generated via simple feature-permutations/perturbations of the training data as in ([Bucila et al, 2006](https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf)) or via a generative model trained on the training data features as in ([Fakoor et al, 2020](https://arxiv.org/abs/2006.14284)).\n",
    "\n",
    "3. Use the stack-ensemble to **predict** on each augmented datapoint and treat these as the corresponding target values. For classification tasks, these are predicted class-probabilities rather than predicted class-labels.\n",
    "\n",
    "4. Merge the original training data and the augmented data into one large dataset and **fit** the single student model to this dataset.\n",
    "\n",
    "In step 2, we typically generate 5-10x as many augmented datapoints as there are training samples, so the student model can better mimic the teacher by being trained to replicate its predictions at a vast number of points in the feature-space. In AutoGluon, this is done as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Distilling with teacher_preds=soft, augment_method=spunge ...\n",
      "SPUNGE: Augmenting training data with 4000 synthetic samples for distillation...\n",
      "Distilling with each of these student models: ['LightGBMClassifier_DSTL']\n",
      "Fitting model: LightGBMClassifier_DSTL ... Training model for up to 120.0s of the 120.0s of remaining time.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's soft_log_loss: -0.821886\tvalid_set's soft_log_loss: -0.866148\n",
      "[100]\ttrain_set's soft_log_loss: -0.810469\tvalid_set's soft_log_loss: -0.853696\n",
      "[150]\ttrain_set's soft_log_loss: -0.806772\tvalid_set's soft_log_loss: -0.847813\n",
      "[200]\ttrain_set's soft_log_loss: -0.804357\tvalid_set's soft_log_loss: -0.844715\n",
      "[250]\ttrain_set's soft_log_loss: -0.802466\tvalid_set's soft_log_loss: -0.844028\n",
      "[300]\ttrain_set's soft_log_loss: -0.800868\tvalid_set's soft_log_loss: -0.841656\n",
      "[350]\ttrain_set's soft_log_loss: -0.79947\tvalid_set's soft_log_loss: -0.840694\n",
      "[400]\ttrain_set's soft_log_loss: -0.798206\tvalid_set's soft_log_loss: -0.839248\n",
      "[450]\ttrain_set's soft_log_loss: -0.79706\tvalid_set's soft_log_loss: -0.837473\n",
      "[500]\ttrain_set's soft_log_loss: -0.795989\tvalid_set's soft_log_loss: -0.836912\n",
      "[550]\ttrain_set's soft_log_loss: -0.795029\tvalid_set's soft_log_loss: -0.836592\n",
      "[600]\ttrain_set's soft_log_loss: -0.794121\tvalid_set's soft_log_loss: -0.835849\n",
      "[650]\ttrain_set's soft_log_loss: -0.793275\tvalid_set's soft_log_loss: -0.835765\n",
      "[700]\ttrain_set's soft_log_loss: -0.792448\tvalid_set's soft_log_loss: -0.836045\n",
      "[750]\ttrain_set's soft_log_loss: -0.791723\tvalid_set's soft_log_loss: -0.834605\n",
      "[800]\ttrain_set's soft_log_loss: -0.791007\tvalid_set's soft_log_loss: -0.834411\n",
      "[850]\ttrain_set's soft_log_loss: -0.790341\tvalid_set's soft_log_loss: -0.83423\n",
      "[900]\ttrain_set's soft_log_loss: -0.78972\tvalid_set's soft_log_loss: -0.833972\n",
      "[950]\ttrain_set's soft_log_loss: -0.789121\tvalid_set's soft_log_loss: -0.833113\n",
      "[1000]\ttrain_set's soft_log_loss: -0.788569\tvalid_set's soft_log_loss: -0.832448\n",
      "[1050]\ttrain_set's soft_log_loss: -0.788033\tvalid_set's soft_log_loss: -0.83202\n",
      "[1100]\ttrain_set's soft_log_loss: -0.787529\tvalid_set's soft_log_loss: -0.83107\n",
      "[1150]\ttrain_set's soft_log_loss: -0.787041\tvalid_set's soft_log_loss: -0.831007\n",
      "[1200]\ttrain_set's soft_log_loss: -0.786565\tvalid_set's soft_log_loss: -0.831291\n",
      "[1250]\ttrain_set's soft_log_loss: -0.78608\tvalid_set's soft_log_loss: -0.830732\n",
      "[1300]\ttrain_set's soft_log_loss: -0.78562\tvalid_set's soft_log_loss: -0.829431\n",
      "[1350]\ttrain_set's soft_log_loss: -0.785173\tvalid_set's soft_log_loss: -0.829412\n",
      "[1400]\ttrain_set's soft_log_loss: -0.784756\tvalid_set's soft_log_loss: -0.829498\n",
      "[1450]\ttrain_set's soft_log_loss: -0.78436\tvalid_set's soft_log_loss: -0.829448\n",
      "[1500]\ttrain_set's soft_log_loss: -0.78396\tvalid_set's soft_log_loss: -0.830148\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t15.52s\t = Training runtime\n",
      "\t0.12s\t = Validation runtime\n",
      "\t0.66\t = Validation accuracy score\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name of distillated model: LightGBMClassifier_DSTL\n"
     ]
    }
   ],
   "source": [
    "student_name = predictor.distill(time_limits=time_limits, hyperparameters={'GBM':{}}, verbosity=3)\n",
    "print(f\"Name of distillated model: {student_name[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, `hyperparameters` allows us to specify which types of models to consider as students for distillation as well as their hyperparameter-values. For instance, if we want really small student models, we can suitably specify the apppropriate hyperparameter-values that affect model size. We can predict with the resulting distilled model or compare its accuracy with the ensemble-models previously trained during `fit()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['NO' 'NO' 'NO' 'NO' '>30' '>30' '>30' 'NO' '>30' '>30' 'NO' 'NO' 'NO'\n",
      " 'NO' 'NO' 'NO' 'NO' '>30' '>30' 'NO']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>score_test</th>\n",
       "      <th>score_val</th>\n",
       "      <th>pred_time_test</th>\n",
       "      <th>pred_time_val</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>pred_time_test_marginal</th>\n",
       "      <th>pred_time_val_marginal</th>\n",
       "      <th>fit_time_marginal</th>\n",
       "      <th>stack_level</th>\n",
       "      <th>can_infer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LightGBMClassifier_STACKER_l1</td>\n",
       "      <td>0.551</td>\n",
       "      <td>0.574</td>\n",
       "      <td>10.829331</td>\n",
       "      <td>8.360810</td>\n",
       "      <td>62.905900</td>\n",
       "      <td>0.386916</td>\n",
       "      <td>0.410340</td>\n",
       "      <td>14.708010</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>weighted_ensemble_k0_l2</td>\n",
       "      <td>0.551</td>\n",
       "      <td>0.574</td>\n",
       "      <td>10.833116</td>\n",
       "      <td>8.363896</td>\n",
       "      <td>63.292353</td>\n",
       "      <td>0.003785</td>\n",
       "      <td>0.003086</td>\n",
       "      <td>0.386453</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CatboostClassifier_STACKER_l0</td>\n",
       "      <td>0.550</td>\n",
       "      <td>0.582</td>\n",
       "      <td>0.178802</td>\n",
       "      <td>0.385099</td>\n",
       "      <td>13.741464</td>\n",
       "      <td>0.178802</td>\n",
       "      <td>0.385099</td>\n",
       "      <td>13.741464</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RandomForestClassifierGini_STACKER_l1</td>\n",
       "      <td>0.542</td>\n",
       "      <td>0.553</td>\n",
       "      <td>11.787190</td>\n",
       "      <td>9.339597</td>\n",
       "      <td>58.561704</td>\n",
       "      <td>1.344775</td>\n",
       "      <td>1.389127</td>\n",
       "      <td>10.363814</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RandomForestClassifierGini_STACKER_l0</td>\n",
       "      <td>0.541</td>\n",
       "      <td>0.554</td>\n",
       "      <td>1.697313</td>\n",
       "      <td>1.277415</td>\n",
       "      <td>6.900042</td>\n",
       "      <td>1.697313</td>\n",
       "      <td>1.277415</td>\n",
       "      <td>6.900042</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ExtraTreesClassifierEntr_STACKER_l0</td>\n",
       "      <td>0.540</td>\n",
       "      <td>0.527</td>\n",
       "      <td>1.969712</td>\n",
       "      <td>1.275319</td>\n",
       "      <td>6.131904</td>\n",
       "      <td>1.969712</td>\n",
       "      <td>1.275319</td>\n",
       "      <td>6.131904</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>LightGBMClassifier_FULL_STACKER_l0</td>\n",
       "      <td>0.539</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.044287</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.349791</td>\n",
       "      <td>0.044287</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.349791</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>RandomForestClassifierEntr_STACKER_l0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>0.553</td>\n",
       "      <td>1.640646</td>\n",
       "      <td>1.279882</td>\n",
       "      <td>7.926347</td>\n",
       "      <td>1.640646</td>\n",
       "      <td>1.279882</td>\n",
       "      <td>7.926347</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ExtraTreesClassifierGini_STACKER_l0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>0.509</td>\n",
       "      <td>2.380385</td>\n",
       "      <td>1.295418</td>\n",
       "      <td>6.231692</td>\n",
       "      <td>2.380385</td>\n",
       "      <td>1.295418</td>\n",
       "      <td>6.231692</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>RandomForestClassifierEntr_STACKER_l1</td>\n",
       "      <td>0.538</td>\n",
       "      <td>0.555</td>\n",
       "      <td>11.862324</td>\n",
       "      <td>9.287851</td>\n",
       "      <td>58.306758</td>\n",
       "      <td>1.419909</td>\n",
       "      <td>1.337381</td>\n",
       "      <td>10.108868</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>ExtraTreesClassifierEntr_STACKER_l1</td>\n",
       "      <td>0.535</td>\n",
       "      <td>0.538</td>\n",
       "      <td>12.222869</td>\n",
       "      <td>9.220299</td>\n",
       "      <td>54.190841</td>\n",
       "      <td>1.780454</td>\n",
       "      <td>1.269828</td>\n",
       "      <td>5.992951</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>ExtraTreesClassifierGini_STACKER_l1</td>\n",
       "      <td>0.535</td>\n",
       "      <td>0.543</td>\n",
       "      <td>12.366913</td>\n",
       "      <td>9.212859</td>\n",
       "      <td>54.446588</td>\n",
       "      <td>1.924498</td>\n",
       "      <td>1.262388</td>\n",
       "      <td>6.248698</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>LightGBMClassifier_STACKER_l0</td>\n",
       "      <td>0.530</td>\n",
       "      <td>0.585</td>\n",
       "      <td>0.332343</td>\n",
       "      <td>0.306638</td>\n",
       "      <td>7.051510</td>\n",
       "      <td>0.332343</td>\n",
       "      <td>0.306638</td>\n",
       "      <td>7.051510</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>weighted_ensemble_k0_l1</td>\n",
       "      <td>0.530</td>\n",
       "      <td>0.585</td>\n",
       "      <td>0.335901</td>\n",
       "      <td>0.311344</td>\n",
       "      <td>7.796096</td>\n",
       "      <td>0.003558</td>\n",
       "      <td>0.004706</td>\n",
       "      <td>0.744586</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>LightGBMClassifier_DSTL</td>\n",
       "      <td>0.526</td>\n",
       "      <td>0.660</td>\n",
       "      <td>0.482685</td>\n",
       "      <td>0.124583</td>\n",
       "      <td>15.521848</td>\n",
       "      <td>0.482685</td>\n",
       "      <td>0.124583</td>\n",
       "      <td>15.521848</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>KNeighborsClassifierDist_STACKER_l1</td>\n",
       "      <td>0.477</td>\n",
       "      <td>0.473</td>\n",
       "      <td>11.666180</td>\n",
       "      <td>9.021663</td>\n",
       "      <td>48.332786</td>\n",
       "      <td>1.223765</td>\n",
       "      <td>1.071193</td>\n",
       "      <td>0.134896</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>KNeighborsClassifierUnif_STACKER_l0</td>\n",
       "      <td>0.468</td>\n",
       "      <td>0.440</td>\n",
       "      <td>1.125564</td>\n",
       "      <td>1.065157</td>\n",
       "      <td>0.109639</td>\n",
       "      <td>1.125564</td>\n",
       "      <td>1.065157</td>\n",
       "      <td>0.109639</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>KNeighborsClassifierUnif_STACKER_l1</td>\n",
       "      <td>0.468</td>\n",
       "      <td>0.452</td>\n",
       "      <td>11.842729</td>\n",
       "      <td>9.001932</td>\n",
       "      <td>48.356426</td>\n",
       "      <td>1.400314</td>\n",
       "      <td>1.051461</td>\n",
       "      <td>0.158536</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>KNeighborsClassifierDist_STACKER_l0</td>\n",
       "      <td>0.467</td>\n",
       "      <td>0.467</td>\n",
       "      <td>1.117650</td>\n",
       "      <td>1.065543</td>\n",
       "      <td>0.105294</td>\n",
       "      <td>1.117650</td>\n",
       "      <td>1.065543</td>\n",
       "      <td>0.105294</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    model  score_test  score_val  \\\n",
       "0           LightGBMClassifier_STACKER_l1       0.551      0.574   \n",
       "1                 weighted_ensemble_k0_l2       0.551      0.574   \n",
       "2           CatboostClassifier_STACKER_l0       0.550      0.582   \n",
       "3   RandomForestClassifierGini_STACKER_l1       0.542      0.553   \n",
       "4   RandomForestClassifierGini_STACKER_l0       0.541      0.554   \n",
       "5     ExtraTreesClassifierEntr_STACKER_l0       0.540      0.527   \n",
       "6      LightGBMClassifier_FULL_STACKER_l0       0.539        NaN   \n",
       "7   RandomForestClassifierEntr_STACKER_l0       0.538      0.553   \n",
       "8     ExtraTreesClassifierGini_STACKER_l0       0.538      0.509   \n",
       "9   RandomForestClassifierEntr_STACKER_l1       0.538      0.555   \n",
       "10    ExtraTreesClassifierEntr_STACKER_l1       0.535      0.538   \n",
       "11    ExtraTreesClassifierGini_STACKER_l1       0.535      0.543   \n",
       "12          LightGBMClassifier_STACKER_l0       0.530      0.585   \n",
       "13                weighted_ensemble_k0_l1       0.530      0.585   \n",
       "14                LightGBMClassifier_DSTL       0.526      0.660   \n",
       "15    KNeighborsClassifierDist_STACKER_l1       0.477      0.473   \n",
       "16    KNeighborsClassifierUnif_STACKER_l0       0.468      0.440   \n",
       "17    KNeighborsClassifierUnif_STACKER_l1       0.468      0.452   \n",
       "18    KNeighborsClassifierDist_STACKER_l0       0.467      0.467   \n",
       "\n",
       "    pred_time_test  pred_time_val   fit_time  pred_time_test_marginal  \\\n",
       "0        10.829331       8.360810  62.905900                 0.386916   \n",
       "1        10.833116       8.363896  63.292353                 0.003785   \n",
       "2         0.178802       0.385099  13.741464                 0.178802   \n",
       "3        11.787190       9.339597  58.561704                 1.344775   \n",
       "4         1.697313       1.277415   6.900042                 1.697313   \n",
       "5         1.969712       1.275319   6.131904                 1.969712   \n",
       "6         0.044287            NaN   0.349791                 0.044287   \n",
       "7         1.640646       1.279882   7.926347                 1.640646   \n",
       "8         2.380385       1.295418   6.231692                 2.380385   \n",
       "9        11.862324       9.287851  58.306758                 1.419909   \n",
       "10       12.222869       9.220299  54.190841                 1.780454   \n",
       "11       12.366913       9.212859  54.446588                 1.924498   \n",
       "12        0.332343       0.306638   7.051510                 0.332343   \n",
       "13        0.335901       0.311344   7.796096                 0.003558   \n",
       "14        0.482685       0.124583  15.521848                 0.482685   \n",
       "15       11.666180       9.021663  48.332786                 1.223765   \n",
       "16        1.125564       1.065157   0.109639                 1.125564   \n",
       "17       11.842729       9.001932  48.356426                 1.400314   \n",
       "18        1.117650       1.065543   0.105294                 1.117650   \n",
       "\n",
       "    pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  \n",
       "0                 0.410340          14.708010            1       True  \n",
       "1                 0.003086           0.386453            2       True  \n",
       "2                 0.385099          13.741464            0       True  \n",
       "3                 1.389127          10.363814            1       True  \n",
       "4                 1.277415           6.900042            0       True  \n",
       "5                 1.275319           6.131904            0       True  \n",
       "6                      NaN           0.349791            0       True  \n",
       "7                 1.279882           7.926347            0       True  \n",
       "8                 1.295418           6.231692            0       True  \n",
       "9                 1.337381          10.108868            1       True  \n",
       "10                1.269828           5.992951            1       True  \n",
       "11                1.262388           6.248698            1       True  \n",
       "12                0.306638           7.051510            0       True  \n",
       "13                0.004706           0.744586            1       True  \n",
       "14                0.124583          15.521848            0       True  \n",
       "15                1.071193           0.134896            1       True  \n",
       "16                1.065157           0.109639            0       True  \n",
       "17                1.051461           0.158536            1       True  \n",
       "18                1.065543           0.105294            0       True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds_student = predictor.predict(test_data[:20], model=student_name[0])\n",
    "print(preds_student)\n",
    "\n",
    "perf = predictor.leaderboard(test_data, silent=True)\n",
    "display(perf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the importance of each Feature\n",
    "\n",
    "Often, high accuracy on some test data may not be enough for us to fully trust that our trained predictor will lead to the right decisions when deployed. \n",
    "We may also want to understand which features the predictor relies on to make these accurate predictions. It is not uncommon that a dataset contains spurious correlations due to how it was collected. For example, radiologists discovered one accurate ML classifier was basing its cancer-diagnoses on the spurious presence of a ruler in tumor images ([Patel, 2017](https://www.thedailybeast.com/why-doctors-arent-afraid-of-better-more-efficient-ai-diagnosing-cancer)).\n",
    "\n",
    "One way to quantify how much each feature individually contributes to an (already-trained) predictor's accuracy is via the method of *permutation-shuffling* ([Parr, 2018](https://explained.ai/rf-importance/)). Here, we randomly permute one column's values (corresponding to measurements of one feature) across the rows of the dataset, and measure the resulting drop in predictive performance when the predictor as asked to predict on the resulting data. The size of this drop is considered the feature's importance-score. This way of scoring feature-importance is often more trustworthy than alternatives that make many approximations or are based on oft-unrealistic assumptions. For our trained AutoGluon stack-ensemble predictor, we can compute these scores like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Computing raw permutation importance for 46 features on weighted_ensemble_k0_l1 ...\n",
      "\t25.7s\t= Expected runtime\n",
      "\t25.41s\t= Actual runtime\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature importance scores:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "diag_1                      0.029\n",
       "medical_specialty           0.012\n",
       "number_inpatient            0.010\n",
       "diag_2                      0.006\n",
       "num_procedures              0.006\n",
       "num_medications             0.003\n",
       "metformin                   0.002\n",
       "admission_type_id           0.002\n",
       "gender                      0.001\n",
       "glipizide                   0.001\n",
       "admission_source_id         0.001\n",
       "number_emergency            0.000\n",
       "chlorpropamide              0.000\n",
       "payer_code                  0.000\n",
       "max_glu_serum               0.000\n",
       "A1Cresult                   0.000\n",
       "weight                      0.000\n",
       "nateglinide                 0.000\n",
       "number_outpatient           0.000\n",
       "diabetesMed                 0.000\n",
       "examide                     0.000\n",
       "troglitazone                0.000\n",
       "metformin-pioglitazone      0.000\n",
       "metformin-rosiglitazone     0.000\n",
       "glimepiride-pioglitazone    0.000\n",
       "glipizide-metformin         0.000\n",
       "glyburide-metformin         0.000\n",
       "citoglipton                 0.000\n",
       "acetohexamide               0.000\n",
       "tolazamide                  0.000\n",
       "repaglinide                 0.000\n",
       "miglitol                    0.000\n",
       "acarbose                    0.000\n",
       "rosiglitazone               0.000\n",
       "pioglitazone                0.000\n",
       "tolbutamide                 0.000\n",
       "glyburide                   0.000\n",
       "insulin                    -0.001\n",
       "discharge_disposition_id   -0.001\n",
       "glimepiride                -0.003\n",
       "change                     -0.005\n",
       "num_lab_procedures         -0.007\n",
       "diag_3                     -0.008\n",
       "number_diagnoses           -0.010\n",
       "time_in_hospital           -0.011\n",
       "age                        -0.011\n",
       "dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "feature_importances = predictor.feature_importance(test_data)\n",
    "print(\"Feature importance scores:\")\n",
    "display(feature_importances)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top features in this list contribute most to AutoGluon's accuracy (for predicting when/if a patient will be readmitted to the hospital).  Features with non-positive importance score hardly contribute to the predictor's accuracy (at least not on an individual basis, these features may theoretically still provide useful predictive signal through interaction-effects with other features' values).\n",
    "\n",
    "If reducing inference latency matters to us, we can also train a more-efficient predictor that only operates on a small subset of the most useful features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>diag_1</th>\n",
       "      <th>medical_specialty</th>\n",
       "      <th>number_inpatient</th>\n",
       "      <th>diag_2</th>\n",
       "      <th>num_procedures</th>\n",
       "      <th>num_medications</th>\n",
       "      <th>metformin</th>\n",
       "      <th>admission_type_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>glipizide</th>\n",
       "      <th>admission_source_id</th>\n",
       "      <th>readmitted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>250.83</td>\n",
       "      <td>Pediatrics-Endocrinology</td>\n",
       "      <td>0</td>\n",
       "      <td>?</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "      <td>\"6\"</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>276</td>\n",
       "      <td>?</td>\n",
       "      <td>0</td>\n",
       "      <td>250.01</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>No</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>&gt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>648</td>\n",
       "      <td>?</td>\n",
       "      <td>1</td>\n",
       "      <td>250</td>\n",
       "      <td>5</td>\n",
       "      <td>13</td>\n",
       "      <td>No</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>Female</td>\n",
       "      <td>Steady</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>?</td>\n",
       "      <td>0</td>\n",
       "      <td>250.43</td>\n",
       "      <td>1</td>\n",
       "      <td>16</td>\n",
       "      <td>No</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>197</td>\n",
       "      <td>?</td>\n",
       "      <td>0</td>\n",
       "      <td>157</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>No</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>Male</td>\n",
       "      <td>Steady</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>38</td>\n",
       "      <td>InternalMedicine</td>\n",
       "      <td>1</td>\n",
       "      <td>788</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>No</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>&lt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>250.13</td>\n",
       "      <td>Pediatrics-CriticalCare</td>\n",
       "      <td>0</td>\n",
       "      <td>?</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>No</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>414</td>\n",
       "      <td>Cardiology</td>\n",
       "      <td>0</td>\n",
       "      <td>411</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>No</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>&gt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>410</td>\n",
       "      <td>Cardiology</td>\n",
       "      <td>0</td>\n",
       "      <td>285</td>\n",
       "      <td>5</td>\n",
       "      <td>24</td>\n",
       "      <td>No</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>&gt;30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>414</td>\n",
       "      <td>Cardiology</td>\n",
       "      <td>0</td>\n",
       "      <td>997</td>\n",
       "      <td>6</td>\n",
       "      <td>26</td>\n",
       "      <td>Steady</td>\n",
       "      <td>\"2\"</td>\n",
       "      <td>Male</td>\n",
       "      <td>Steady</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>NO</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     diag_1         medical_specialty  number_inpatient  diag_2  \\\n",
       "0    250.83  Pediatrics-Endocrinology                 0       ?   \n",
       "1       276                         ?                 0  250.01   \n",
       "2       648                         ?                 1     250   \n",
       "3         8                         ?                 0  250.43   \n",
       "4       197                         ?                 0     157   \n",
       "..      ...                       ...               ...     ...   \n",
       "995      38          InternalMedicine                 1     788   \n",
       "996  250.13   Pediatrics-CriticalCare                 0       ?   \n",
       "997     414                Cardiology                 0     411   \n",
       "998     410                Cardiology                 0     285   \n",
       "999     414                Cardiology                 0     997   \n",
       "\n",
       "     num_procedures  num_medications metformin admission_type_id  gender  \\\n",
       "0                 0                1        No               \"6\"  Female   \n",
       "1                 0               18        No               \"1\"  Female   \n",
       "2                 5               13        No               \"1\"  Female   \n",
       "3                 1               16        No               \"1\"    Male   \n",
       "4                 0                8        No               \"1\"    Male   \n",
       "..              ...              ...       ...               ...     ...   \n",
       "995               1               19        No               \"1\"    Male   \n",
       "996               0                8        No               \"2\"    Male   \n",
       "997               0               16        No               \"1\"    Male   \n",
       "998               5               24        No               \"1\"  Female   \n",
       "999               6               26    Steady               \"2\"    Male   \n",
       "\n",
       "    glipizide admission_source_id readmitted  \n",
       "0          No                 \"1\"         NO  \n",
       "1          No                 \"7\"        >30  \n",
       "2      Steady                 \"7\"         NO  \n",
       "3          No                 \"7\"         NO  \n",
       "4      Steady                 \"7\"         NO  \n",
       "..        ...                 ...        ...  \n",
       "995        No                 \"7\"        <30  \n",
       "996        No                 \"1\"         NO  \n",
       "997        No                 \"7\"        >30  \n",
       "998        No                 \"7\"        >30  \n",
       "999    Steady                 \"1\"         NO  \n",
       "\n",
       "[1000 rows x 12 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No output_directory specified. Models will be saved in: AutogluonModels/ag-20200801_083345/\n",
      "Beginning AutoGluon training ... Time limit = 120s\n",
      "AutoGluon will save models to AutogluonModels/ag-20200801_083345/\n",
      "AutoGluon Version:  0.0.13b20200731\n",
      "Train Data Rows:    1000\n",
      "Train Data Columns: 12\n",
      "Preprocessing data ...\n",
      "Here are the 3 unique label values in your data:  ['NO', '>30', '<30']\n",
      "AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object).\n",
      "If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "\n",
      "Train Data Class Count: 3\n",
      "Feature Generator processed 1000 data points with 11 features\n",
      "Original Features (raw dtypes):\n",
      "\tobject features: 8\n",
      "\tint64 features: 3\n",
      "Original Features (inferred dtypes):\n",
      "\tobject features: 8\n",
      "\tint features: 3\n",
      "Generated Features (special dtypes):\n",
      "Processed Features (raw dtypes):\n",
      "\tint features: 3\n",
      "\tcategory features: 8\n",
      "Processed Features:\n",
      "\tint features: 3\n",
      "\tcategory features: 8\n",
      "\tData preprocessing and feature engineering runtime = 0.2s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: accuracy\n",
      "To change this, specify the eval_metric argument of fit()\n",
      "AutoGluon will early stop models using evaluation metric: accuracy\n",
      "Fitting model: RandomForestClassifierGini ... Training model for up to 119.8s of the 119.8s of remaining time.\n",
      "\t0.535\t = Validation accuracy score\n",
      "\t0.89s\t = Training runtime\n",
      "\t0.12s\t = Validation runtime\n",
      "Fitting model: RandomForestClassifierEntr ... Training model for up to 118.72s of the 118.72s of remaining time.\n",
      "\t0.525\t = Validation accuracy score\n",
      "\t1.06s\t = Training runtime\n",
      "\t0.12s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierGini ... Training model for up to 117.44s of the 117.44s of remaining time.\n",
      "\t0.525\t = Validation accuracy score\n",
      "\t0.84s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierEntr ... Training model for up to 116.31s of the 116.31s of remaining time.\n",
      "\t0.52\t = Validation accuracy score\n",
      "\t0.78s\t = Training runtime\n",
      "\t0.12s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierUnif ... Training model for up to 115.29s of the 115.29s of remaining time.\n",
      "\t0.495\t = Validation accuracy score\n",
      "\t0.0s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: KNeighborsClassifierDist ... Training model for up to 115.17s of the 115.17s of remaining time.\n",
      "\t0.455\t = Validation accuracy score\n",
      "\t0.0s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifier ... Training model for up to 115.04s of the 115.04s of remaining time.\n",
      "\t0.62\t = Validation accuracy score\n",
      "\t2.23s\t = Training runtime\n",
      "\t0.01s\t = Validation runtime\n",
      "Fitting model: CatboostClassifier ... Training model for up to 112.79s of the 112.79s of remaining time.\n",
      "\t0.635\t = Validation accuracy score\n",
      "\t2.01s\t = Training runtime\n",
      "\t0.01s\t = Validation runtime\n",
      "Fitting model: NeuralNetClassifier ... Training model for up to 110.77s of the 110.76s of remaining time.\n",
      "\t0.54\t = Validation accuracy score\n",
      "\t9.29s\t = Training runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifierCustom ... Training model for up to 101.41s of the 101.41s of remaining time.\n",
      "\t0.59\t = Validation accuracy score\n",
      "\t2.71s\t = Training runtime\n",
      "\t0.02s\t = Validation runtime\n",
      "Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 119.8s of the 97.24s of remaining time.\n",
      "\t0.635\t = Validation accuracy score\n",
      "\t0.42s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 23.25s ...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                         model  score_test  score_val  pred_time_test  pred_time_val  fit_time  pred_time_test_marginal  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer\n",
      "0           LightGBMClassifier       0.546      0.620        0.033110       0.013984  2.231284                 0.033110                0.013984           2.231284            0       True\n",
      "1           CatboostClassifier       0.533      0.635        0.012914       0.009126  2.005353                 0.012914                0.009126           2.005353            0       True\n",
      "2      weighted_ensemble_k0_l1       0.533      0.635        0.018172       0.010346  2.424684                 0.005258                0.001220           0.419331            1       True\n",
      "3   RandomForestClassifierEntr       0.519      0.525        0.126994       0.119581  1.055600                 0.126994                0.119581           1.055600            0       True\n",
      "4   RandomForestClassifierGini       0.513      0.535        0.127619       0.123259  0.893335                 0.127619                0.123259           0.893335            0       True\n",
      "5     LightGBMClassifierCustom       0.484      0.590        0.162309       0.018341  2.712698                 0.162309                0.018341           2.712698            0       True\n",
      "6     ExtraTreesClassifierGini       0.483      0.525        0.163947       0.132366  0.842483                 0.163947                0.132366           0.842483            0       True\n",
      "7     ExtraTreesClassifierEntr       0.482      0.520        0.176688       0.116681  0.776156                 0.176688                0.116681           0.776156            0       True\n",
      "8     KNeighborsClassifierUnif       0.456      0.495        0.107943       0.107785  0.003167                 0.107943                0.107785           0.003167            0       True\n",
      "9     KNeighborsClassifierDist       0.436      0.455        0.110001       0.114218  0.003697                 0.110001                0.114218           0.003697            0       True\n",
      "10         NeuralNetClassifier       0.399      0.540        0.172961       0.038615  9.288040                 0.172961                0.038615           9.288040            0       True\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>score_test</th>\n",
       "      <th>score_val</th>\n",
       "      <th>pred_time_test</th>\n",
       "      <th>pred_time_val</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>pred_time_test_marginal</th>\n",
       "      <th>pred_time_val_marginal</th>\n",
       "      <th>fit_time_marginal</th>\n",
       "      <th>stack_level</th>\n",
       "      <th>can_infer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LightGBMClassifier</td>\n",
       "      <td>0.546</td>\n",
       "      <td>0.620</td>\n",
       "      <td>0.033110</td>\n",
       "      <td>0.013984</td>\n",
       "      <td>2.231284</td>\n",
       "      <td>0.033110</td>\n",
       "      <td>0.013984</td>\n",
       "      <td>2.231284</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CatboostClassifier</td>\n",
       "      <td>0.533</td>\n",
       "      <td>0.635</td>\n",
       "      <td>0.012914</td>\n",
       "      <td>0.009126</td>\n",
       "      <td>2.005353</td>\n",
       "      <td>0.012914</td>\n",
       "      <td>0.009126</td>\n",
       "      <td>2.005353</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>weighted_ensemble_k0_l1</td>\n",
       "      <td>0.533</td>\n",
       "      <td>0.635</td>\n",
       "      <td>0.018172</td>\n",
       "      <td>0.010346</td>\n",
       "      <td>2.424684</td>\n",
       "      <td>0.005258</td>\n",
       "      <td>0.001220</td>\n",
       "      <td>0.419331</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RandomForestClassifierEntr</td>\n",
       "      <td>0.519</td>\n",
       "      <td>0.525</td>\n",
       "      <td>0.126994</td>\n",
       "      <td>0.119581</td>\n",
       "      <td>1.055600</td>\n",
       "      <td>0.126994</td>\n",
       "      <td>0.119581</td>\n",
       "      <td>1.055600</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RandomForestClassifierGini</td>\n",
       "      <td>0.513</td>\n",
       "      <td>0.535</td>\n",
       "      <td>0.127619</td>\n",
       "      <td>0.123259</td>\n",
       "      <td>0.893335</td>\n",
       "      <td>0.127619</td>\n",
       "      <td>0.123259</td>\n",
       "      <td>0.893335</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>LightGBMClassifierCustom</td>\n",
       "      <td>0.484</td>\n",
       "      <td>0.590</td>\n",
       "      <td>0.162309</td>\n",
       "      <td>0.018341</td>\n",
       "      <td>2.712698</td>\n",
       "      <td>0.162309</td>\n",
       "      <td>0.018341</td>\n",
       "      <td>2.712698</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ExtraTreesClassifierGini</td>\n",
       "      <td>0.483</td>\n",
       "      <td>0.525</td>\n",
       "      <td>0.163947</td>\n",
       "      <td>0.132366</td>\n",
       "      <td>0.842483</td>\n",
       "      <td>0.163947</td>\n",
       "      <td>0.132366</td>\n",
       "      <td>0.842483</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ExtraTreesClassifierEntr</td>\n",
       "      <td>0.482</td>\n",
       "      <td>0.520</td>\n",
       "      <td>0.176688</td>\n",
       "      <td>0.116681</td>\n",
       "      <td>0.776156</td>\n",
       "      <td>0.176688</td>\n",
       "      <td>0.116681</td>\n",
       "      <td>0.776156</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>KNeighborsClassifierUnif</td>\n",
       "      <td>0.456</td>\n",
       "      <td>0.495</td>\n",
       "      <td>0.107943</td>\n",
       "      <td>0.107785</td>\n",
       "      <td>0.003167</td>\n",
       "      <td>0.107943</td>\n",
       "      <td>0.107785</td>\n",
       "      <td>0.003167</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>KNeighborsClassifierDist</td>\n",
       "      <td>0.436</td>\n",
       "      <td>0.455</td>\n",
       "      <td>0.110001</td>\n",
       "      <td>0.114218</td>\n",
       "      <td>0.003697</td>\n",
       "      <td>0.110001</td>\n",
       "      <td>0.114218</td>\n",
       "      <td>0.003697</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>NeuralNetClassifier</td>\n",
       "      <td>0.399</td>\n",
       "      <td>0.540</td>\n",
       "      <td>0.172961</td>\n",
       "      <td>0.038615</td>\n",
       "      <td>9.288040</td>\n",
       "      <td>0.172961</td>\n",
       "      <td>0.038615</td>\n",
       "      <td>9.288040</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         model  score_test  score_val  pred_time_test  \\\n",
       "0           LightGBMClassifier       0.546      0.620        0.033110   \n",
       "1           CatboostClassifier       0.533      0.635        0.012914   \n",
       "2      weighted_ensemble_k0_l1       0.533      0.635        0.018172   \n",
       "3   RandomForestClassifierEntr       0.519      0.525        0.126994   \n",
       "4   RandomForestClassifierGini       0.513      0.535        0.127619   \n",
       "5     LightGBMClassifierCustom       0.484      0.590        0.162309   \n",
       "6     ExtraTreesClassifierGini       0.483      0.525        0.163947   \n",
       "7     ExtraTreesClassifierEntr       0.482      0.520        0.176688   \n",
       "8     KNeighborsClassifierUnif       0.456      0.495        0.107943   \n",
       "9     KNeighborsClassifierDist       0.436      0.455        0.110001   \n",
       "10         NeuralNetClassifier       0.399      0.540        0.172961   \n",
       "\n",
       "    pred_time_val  fit_time  pred_time_test_marginal  pred_time_val_marginal  \\\n",
       "0        0.013984  2.231284                 0.033110                0.013984   \n",
       "1        0.009126  2.005353                 0.012914                0.009126   \n",
       "2        0.010346  2.424684                 0.005258                0.001220   \n",
       "3        0.119581  1.055600                 0.126994                0.119581   \n",
       "4        0.123259  0.893335                 0.127619                0.123259   \n",
       "5        0.018341  2.712698                 0.162309                0.018341   \n",
       "6        0.132366  0.842483                 0.163947                0.132366   \n",
       "7        0.116681  0.776156                 0.176688                0.116681   \n",
       "8        0.107785  0.003167                 0.107943                0.107785   \n",
       "9        0.114218  0.003697                 0.110001                0.114218   \n",
       "10       0.038615  9.288040                 0.172961                0.038615   \n",
       "\n",
       "    fit_time_marginal  stack_level  can_infer  \n",
       "0            2.231284            0       True  \n",
       "1            2.005353            0       True  \n",
       "2            0.419331            1       True  \n",
       "3            1.055600            0       True  \n",
       "4            0.893335            0       True  \n",
       "5            2.712698            0       True  \n",
       "6            0.842483            0       True  \n",
       "7            0.776156            0       True  \n",
       "8            0.003167            0       True  \n",
       "9            0.003697            0       True  \n",
       "10           9.288040            0       True  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_subset = list(feature_importances[feature_importances > 0].index) + [label_column]\n",
    "train_data_small = train_data[feature_subset]\n",
    "display(train_data_small)\n",
    "\n",
    "predictor_small = task.fit(train_data=train_data_small, label=label_column, time_limits=time_limits)\n",
    "predictor_small.leaderboard(test_data[feature_subset])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practical ML Deployment  with AutoGluon\n",
    "\n",
    "Here's a practical guide on how to use AutoGluon to easily turn raw tabular data into deployed models for applications where inference-latency/costs matter. For demonstration, we'll consider a setting where the predictions will be made on one datapoint at a time (a common pattern in settings where inference-latency really matters). For such settings, here is how we recommend you train with AutoGluon:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Beginning AutoGluon training ... Time limit = 120s\n",
      "AutoGluon will save models to agModels/\n",
      "AutoGluon Version:  0.0.13b20200731\n",
      "Train Data Rows:    1000\n",
      "Train Data Columns: 47\n",
      "Preprocessing data ...\n",
      "Here are the 3 unique label values in your data:  ['NO', '>30', '<30']\n",
      "AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object).\n",
      "If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "\n",
      "Train Data Class Count: 3\n",
      "Feature Generator processed 1000 data points with 33 features\n",
      "Original Features (raw dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat64 features: 1\n",
      "\tint64 features: 7\n",
      "Original Features (inferred dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "Generated Features (special dtypes):\n",
      "Processed Features (raw dtypes):\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "Processed Features:\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "\tData preprocessing and feature engineering runtime = 0.23s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: accuracy\n",
      "To change this, specify the eval_metric argument of fit()\n",
      "AutoGluon will early stop models using evaluation metric: accuracy\n",
      "Fitting model: RandomForestClassifierGini_STACKER_l0 ... Training model for up to 59.89s of the 119.77s of remaining time.\n",
      "\t0.526\t = Validation accuracy score\n",
      "\t8.0s\t = Training runtime\n",
      "\t1.3s\t = Validation runtime\n",
      "Fitting model: RandomForestClassifierEntr_STACKER_l0 ... Training model for up to 50.54s of the 110.42s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 296 due to low time. Expected time usage reduced from 4.1s -> 4.0s...\n",
      "\t0.54\t = Validation accuracy score\n",
      "\t6.93s\t = Training runtime\n",
      "\t1.25s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierGini_STACKER_l0 ... Training model for up to 42.31s of the 102.2s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 245 due to low time. Expected time usage reduced from 4.1s -> 3.4s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 276 due to low time. Expected time usage reduced from 4.0s -> 3.7s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 298 due to low time. Expected time usage reduced from 4.1s -> 4.1s...\n",
      "\t0.52\t = Validation accuracy score\n",
      "\t5.49s\t = Training runtime\n",
      "\t1.24s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierEntr_STACKER_l0 ... Training model for up to 35.52s of the 95.41s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 206 due to low time. Expected time usage reduced from 4.1s -> 2.8s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 227 due to low time. Expected time usage reduced from 4.1s -> 3.1s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 254 due to low time. Expected time usage reduced from 4.1s -> 3.4s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 287 due to low time. Expected time usage reduced from 4.0s -> 3.9s...\n",
      "\t0.524\t = Validation accuracy score\n",
      "\t5.16s\t = Training runtime\n",
      "\t1.27s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifier_STACKER_l0 ... Training model for up to 29.02s of the 88.9s of remaining time.\n",
      "\t0.585\t = Validation accuracy score\n",
      "\t5.04s\t = Training runtime\n",
      "\t0.27s\t = Validation runtime\n",
      "Fitting model: CatboostClassifier_STACKER_l0 ... Training model for up to 23.67s of the 83.56s of remaining time.\n",
      "\tTime limit exceeded... Skipping CatboostClassifier_STACKER_l0.\n",
      "Fitting model: NeuralNetClassifier_STACKER_l0 ... Training model for up to 1.06s of the 60.94s of remaining time.\n",
      "\tRan out of time, stopping training early.\n",
      "\tTime limit exceeded... Skipping NeuralNetClassifier_STACKER_l0.\n",
      "Fitting model: LightGBMClassifierCustom_STACKER_l0 ... Training model for up to 0.61s of the 60.5s of remaining time.\n",
      "\tRan out of time, early stopping on iteration 1. Best iteration is:\n",
      "\t[1]\ttrain_set's multi_error: 0.512222\tvalid_set's multi_error: 0.51\n",
      "\tTime limit exceeded... Skipping LightGBMClassifierCustom_STACKER_l0.\n",
      "Completed 1/20 k-fold bagging repeats ...\n",
      "Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 119.77s of the 60.32s of remaining time.\n",
      "\t0.585\t = Validation accuracy score\n",
      "\t0.27s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Fitting model: RandomForestClassifierGini_STACKER_l1 ... Training model for up to 60.02s of the 60.01s of remaining time.\n",
      "\t0.551\t = Validation accuracy score\n",
      "\t7.74s\t = Training runtime\n",
      "\t1.29s\t = Validation runtime\n",
      "Fitting model: RandomForestClassifierEntr_STACKER_l1 ... Training model for up to 50.94s of the 50.93s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 288 due to low time. Expected time usage reduced from 4.2s -> 4.1s...\n",
      "\t0.557\t = Validation accuracy score\n",
      "\t8.12s\t = Training runtime\n",
      "\t1.28s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierGini_STACKER_l1 ... Training model for up to 41.48s of the 41.47s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 240 due to low time. Expected time usage reduced from 4.1s -> 3.3s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 272 due to low time. Expected time usage reduced from 4.0s -> 3.6s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 296 due to low time. Expected time usage reduced from 4.1s -> 4.0s...\n",
      "\t0.552\t = Validation accuracy score\n",
      "\t5.91s\t = Training runtime\n",
      "\t1.25s\t = Validation runtime\n",
      "Fitting model: ExtraTreesClassifierEntr_STACKER_l1 ... Training model for up to 34.27s of the 34.25s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 199 due to low time. Expected time usage reduced from 4.1s -> 2.7s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 221 due to low time. Expected time usage reduced from 4.0s -> 3.0s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 248 due to low time. Expected time usage reduced from 4.0s -> 3.3s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 274 due to low time. Expected time usage reduced from 4.0s -> 3.7s...\n",
      "\t0.543\t = Validation accuracy score\n",
      "\t5.92s\t = Training runtime\n",
      "\t1.31s\t = Validation runtime\n",
      "Fitting model: LightGBMClassifier_STACKER_l1 ... Training model for up to 26.96s of the 26.95s of remaining time.\n",
      "\t0.589\t = Validation accuracy score\n",
      "\t10.17s\t = Training runtime\n",
      "\t0.31s\t = Validation runtime\n",
      "Fitting model: CatboostClassifier_STACKER_l1 ... Training model for up to 16.45s of the 16.44s of remaining time.\n",
      "\t0.599\t = Validation accuracy score\n",
      "\t15.85s\t = Training runtime\n",
      "\t0.24s\t = Validation runtime\n",
      "Fitting model: NeuralNetClassifier_STACKER_l1 ... Training model for up to 0.3s of the 0.29s of remaining time.\n",
      "\tRan out of time, stopping training early.\n",
      "\tTime limit exceeded... Skipping NeuralNetClassifier_STACKER_l1.\n",
      "Completed 1/20 k-fold bagging repeats ...\n",
      "Fitting model: weighted_ensemble_k0_l2 ... Training model for up to 119.77s of the -0.3s of remaining time.\n",
      "\t0.599\t = Validation accuracy score\n",
      "\t0.36s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 120.69s ...\n",
      "Fitting model: RandomForestClassifierGini_FULL_STACKER_l0 ...\n",
      "\t0.77s\t = Training runtime\n",
      "Fitting model: RandomForestClassifierEntr_FULL_STACKER_l0 ...\n",
      "\t0.8s\t = Training runtime\n",
      "Fitting model: ExtraTreesClassifierGini_FULL_STACKER_l0 ...\n",
      "\t0.65s\t = Training runtime\n",
      "Fitting model: ExtraTreesClassifierEntr_FULL_STACKER_l0 ...\n",
      "\t0.55s\t = Training runtime\n",
      "Fitting model: LightGBMClassifier_FULL_STACKER_l0 ...\n",
      "\t0.17s\t = Training runtime\n",
      "Fitting model: CatboostClassifier_FULL_STACKER_l1 ...\n",
      "\t0.42s\t = Training runtime\n",
      "Deleting model RandomForestClassifierGini_STACKER_l0. All files under agModels/models/RandomForestClassifierGini_STACKER_l0/ will be removed.\n",
      "Deleting model RandomForestClassifierEntr_STACKER_l0. All files under agModels/models/RandomForestClassifierEntr_STACKER_l0/ will be removed.\n",
      "Deleting model ExtraTreesClassifierGini_STACKER_l0. All files under agModels/models/ExtraTreesClassifierGini_STACKER_l0/ will be removed.\n",
      "Deleting model ExtraTreesClassifierEntr_STACKER_l0. All files under agModels/models/ExtraTreesClassifierEntr_STACKER_l0/ will be removed.\n",
      "Deleting model LightGBMClassifier_STACKER_l0. All files under agModels/models/LightGBMClassifier_STACKER_l0/ will be removed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Deleting model weighted_ensemble_k0_l1. All files under agModels/models/weighted_ensemble_k0_l1/ will be removed.\n",
      "Deleting model RandomForestClassifierGini_STACKER_l1. All files under agModels/models/RandomForestClassifierGini_STACKER_l1/ will be removed.\n",
      "Deleting model RandomForestClassifierEntr_STACKER_l1. All files under agModels/models/RandomForestClassifierEntr_STACKER_l1/ will be removed.\n",
      "Deleting model ExtraTreesClassifierGini_STACKER_l1. All files under agModels/models/ExtraTreesClassifierGini_STACKER_l1/ will be removed.\n",
      "Deleting model ExtraTreesClassifierEntr_STACKER_l1. All files under agModels/models/ExtraTreesClassifierEntr_STACKER_l1/ will be removed.\n",
      "Deleting model LightGBMClassifier_STACKER_l1. All files under agModels/models/LightGBMClassifier_STACKER_l1/ will be removed.\n",
      "Deleting model CatboostClassifier_STACKER_l1. All files under agModels/models/CatboostClassifier_STACKER_l1/ will be removed.\n",
      "Deleting model weighted_ensemble_k0_l2. All files under agModels/models/weighted_ensemble_k0_l2/ will be removed.\n"
     ]
    }
   ],
   "source": [
    "model_folder = 'agModels'\n",
    "predictor = task.fit(train_data=train_data, label=label_column, time_limits=time_limits, output_directory=model_folder,\n",
    "                     presets=['good_quality_faster_inference_only_refit', 'optimize_for_deployment'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that AutoGluon will save its models to the **agModels/** folder, which we can then easily move to another server and reload the predictor for deployments. We used the [`presets` argument](https://autogluon.mxnet.io/api/autogluon.task.html#autogluon.task.TabularPrediction.fit) which offers an easy interface to tell AutoGluon whether you care more about accuracy above all else, or whether computational considerations like inference-latency/memory-footprint matter in your applications.\n",
    "For this demonstration, we specified two presets:\n",
    "- **good_quality_faster_inference_only_refit**: tells AutoGluon we care about fast inference\n",
    "- **optimize_for_deployment**: deletes all additional files beyond the key model-files needed to produce predictions. With this preset, some functionality in the `Predictor` may no longer be available, but the key `load()`, `predict()`, `predict_proba()` methods will still work.\n",
    "\n",
    "If you require even lower latency, we recommend additionally specifying the [`fit()` argument](https://autogluon.mxnet.io/api/autogluon.task.html#autogluon.task.TabularPrediction.fit) `hyperparameters = 'light'` or `= 'very_light`. AutoGluon will then train models with hyperparameter-settings that improve their efficiency. You may also manually specify such hyperparameter-settings yourself and also try [`distill()`](https://autogluon.mxnet.io/api/autogluon.task.html#autogluon.task.tabular_prediction.TabularPredictor.distill).\n",
    "\n",
    "Here's how one can deploy the trained predictor on a new server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'path': 'agModels/',\n",
       " 'label': 'readmitted',\n",
       " 'time_fit_preprocessing': 0.22698187828063965,\n",
       " 'time_fit_training': None,\n",
       " 'time_fit_total': None,\n",
       " 'time_limit': 120,\n",
       " 'random_seed': 0,\n",
       " 'version': '0.0.13b20200731',\n",
       " 'time_train_start': 1596270850.380117,\n",
       " 'num_rows_train': 1000,\n",
       " 'num_cols_train': 33,\n",
       " 'num_classes': 3,\n",
       " 'problem_type': 'multiclass',\n",
       " 'eval_metric': 'accuracy',\n",
       " 'stopping_metric': 'accuracy',\n",
       " 'best_model': 'CatboostClassifier_FULL_STACKER_l1',\n",
       " 'best_model_score_val': None,\n",
       " 'best_model_stack_level': 1,\n",
       " 'num_models_trained': 6,\n",
       " 'num_bagging_folds': 10,\n",
       " 'max_stack_level': 2,\n",
       " 'max_core_stack_level': 1,\n",
       " 'model_stack_info': defaultdict(<function autogluon.utils.tabular.ml.utils.dd_list()>,\n",
       "             {'core': defaultdict(list, {0: [], 1: []}),\n",
       "              'aux1': defaultdict(list, {1: [], 2: []}),\n",
       "              'refit_single_full': defaultdict(list,\n",
       "                          {0: ['RandomForestClassifierGini_FULL_STACKER_l0',\n",
       "                            'RandomForestClassifierEntr_FULL_STACKER_l0',\n",
       "                            'ExtraTreesClassifierGini_FULL_STACKER_l0',\n",
       "                            'ExtraTreesClassifierEntr_FULL_STACKER_l0',\n",
       "                            'LightGBMClassifier_FULL_STACKER_l0'],\n",
       "                           1: ['CatboostClassifier_FULL_STACKER_l1']})}),\n",
       " 'model_info': {'RandomForestClassifierGini_FULL_STACKER_l0': {'name': 'RandomForestClassifierGini_FULL_STACKER_l0',\n",
       "   'model_type': 'StackerEnsembleModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.7719159126281738,\n",
       "   'predict_time': None,\n",
       "   'val_score': None,\n",
       "   'hyperparameters': {'max_models': 25, 'max_models_per_type': 5},\n",
       "   'hyperparameters_fit': {},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 11858994,\n",
       "   'bagged_info': {'child_type': 'RFModel',\n",
       "    'num_child_models': 1,\n",
       "    'child_model_names': ['RandomForestClassifierGini_FULL'],\n",
       "    '_n_repeats': 1,\n",
       "    '_k_per_n_repeat': [1],\n",
       "    '_random_state': 0,\n",
       "    'low_memory': True,\n",
       "    'bagged_mode': False,\n",
       "    'max_memory_size': 23716252,\n",
       "    'min_memory_size': 23716252},\n",
       "   'stacker_info': {'num_base_models': 0,\n",
       "    'base_model_names': [],\n",
       "    'use_orig_features': True},\n",
       "   'children_info': {'RandomForestClassifierGini_FULL': {'name': 'RandomForestClassifierGini_FULL',\n",
       "     'model_type': 'RFModel',\n",
       "     'problem_type': 'multiclass',\n",
       "     'eval_metric': 'accuracy',\n",
       "     'stopping_metric': 'accuracy',\n",
       "     'fit_time': 0.7719159126281738,\n",
       "     'predict_time': None,\n",
       "     'val_score': None,\n",
       "     'hyperparameters': {'n_estimators': 300,\n",
       "      'n_jobs': -1,\n",
       "      'criterion': 'gini',\n",
       "      'max_depth': 15},\n",
       "     'hyperparameters_fit': {'n_estimators': 300},\n",
       "     'hyperparameters_nondefault': ['criterion', 'max_depth'],\n",
       "     'memory_size': 11857258}}},\n",
       "  'RandomForestClassifierEntr_FULL_STACKER_l0': {'name': 'RandomForestClassifierEntr_FULL_STACKER_l0',\n",
       "   'model_type': 'StackerEnsembleModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.7970209121704102,\n",
       "   'predict_time': None,\n",
       "   'val_score': None,\n",
       "   'hyperparameters': {'max_models': 25, 'max_models_per_type': 5},\n",
       "   'hyperparameters_fit': {},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 11581240,\n",
       "   'bagged_info': {'child_type': 'RFModel',\n",
       "    'num_child_models': 1,\n",
       "    'child_model_names': ['RandomForestClassifierEntr_FULL'],\n",
       "    '_n_repeats': 1,\n",
       "    '_k_per_n_repeat': [1],\n",
       "    '_random_state': 0,\n",
       "    'low_memory': True,\n",
       "    'bagged_mode': False,\n",
       "    'max_memory_size': 23160744,\n",
       "    'min_memory_size': 23160744},\n",
       "   'stacker_info': {'num_base_models': 0,\n",
       "    'base_model_names': [],\n",
       "    'use_orig_features': True},\n",
       "   'children_info': {'RandomForestClassifierEntr_FULL': {'name': 'RandomForestClassifierEntr_FULL',\n",
       "     'model_type': 'RFModel',\n",
       "     'problem_type': 'multiclass',\n",
       "     'eval_metric': 'accuracy',\n",
       "     'stopping_metric': 'accuracy',\n",
       "     'fit_time': 0.7970209121704102,\n",
       "     'predict_time': None,\n",
       "     'val_score': None,\n",
       "     'hyperparameters': {'n_estimators': 300,\n",
       "      'n_jobs': -1,\n",
       "      'criterion': 'entropy',\n",
       "      'max_depth': 15},\n",
       "     'hyperparameters_fit': {'n_estimators': 300},\n",
       "     'hyperparameters_nondefault': ['criterion', 'max_depth'],\n",
       "     'memory_size': 11579504}}},\n",
       "  'ExtraTreesClassifierGini_FULL_STACKER_l0': {'name': 'ExtraTreesClassifierGini_FULL_STACKER_l0',\n",
       "   'model_type': 'StackerEnsembleModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.6507630348205566,\n",
       "   'predict_time': None,\n",
       "   'val_score': None,\n",
       "   'hyperparameters': {'max_models': 25, 'max_models_per_type': 5},\n",
       "   'hyperparameters_fit': {},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 16195046,\n",
       "   'bagged_info': {'child_type': 'XTModel',\n",
       "    'num_child_models': 1,\n",
       "    'child_model_names': ['ExtraTreesClassifierGini_FULL'],\n",
       "    '_n_repeats': 1,\n",
       "    '_k_per_n_repeat': [1],\n",
       "    '_random_state': 0,\n",
       "    'low_memory': True,\n",
       "    'bagged_mode': False,\n",
       "    'max_memory_size': 32388362,\n",
       "    'min_memory_size': 32388362},\n",
       "   'stacker_info': {'num_base_models': 0,\n",
       "    'base_model_names': [],\n",
       "    'use_orig_features': True},\n",
       "   'children_info': {'ExtraTreesClassifierGini_FULL': {'name': 'ExtraTreesClassifierGini_FULL',\n",
       "     'model_type': 'XTModel',\n",
       "     'problem_type': 'multiclass',\n",
       "     'eval_metric': 'accuracy',\n",
       "     'stopping_metric': 'accuracy',\n",
       "     'fit_time': 0.6507630348205566,\n",
       "     'predict_time': None,\n",
       "     'val_score': None,\n",
       "     'hyperparameters': {'n_estimators': 292,\n",
       "      'n_jobs': -1,\n",
       "      'criterion': 'gini',\n",
       "      'max_depth': 15},\n",
       "     'hyperparameters_fit': {'n_estimators': 292},\n",
       "     'hyperparameters_nondefault': ['criterion', 'max_depth'],\n",
       "     'memory_size': 16193316}}},\n",
       "  'ExtraTreesClassifierEntr_FULL_STACKER_l0': {'name': 'ExtraTreesClassifierEntr_FULL_STACKER_l0',\n",
       "   'model_type': 'StackerEnsembleModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.5474319458007812,\n",
       "   'predict_time': None,\n",
       "   'val_score': None,\n",
       "   'hyperparameters': {'max_models': 25, 'max_models_per_type': 5},\n",
       "   'hyperparameters_fit': {},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 15928071,\n",
       "   'bagged_info': {'child_type': 'XTModel',\n",
       "    'num_child_models': 1,\n",
       "    'child_model_names': ['ExtraTreesClassifierEntr_FULL'],\n",
       "    '_n_repeats': 1,\n",
       "    '_k_per_n_repeat': [1],\n",
       "    '_random_state': 0,\n",
       "    'low_memory': True,\n",
       "    'bagged_mode': False,\n",
       "    'max_memory_size': 31854412,\n",
       "    'min_memory_size': 31854412},\n",
       "   'stacker_info': {'num_base_models': 0,\n",
       "    'base_model_names': [],\n",
       "    'use_orig_features': True},\n",
       "   'children_info': {'ExtraTreesClassifierEntr_FULL': {'name': 'ExtraTreesClassifierEntr_FULL',\n",
       "     'model_type': 'XTModel',\n",
       "     'problem_type': 'multiclass',\n",
       "     'eval_metric': 'accuracy',\n",
       "     'stopping_metric': 'accuracy',\n",
       "     'fit_time': 0.5474319458007812,\n",
       "     'predict_time': None,\n",
       "     'val_score': None,\n",
       "     'hyperparameters': {'n_estimators': 277,\n",
       "      'n_jobs': -1,\n",
       "      'criterion': 'entropy',\n",
       "      'max_depth': 15},\n",
       "     'hyperparameters_fit': {'n_estimators': 277},\n",
       "     'hyperparameters_nondefault': ['criterion', 'max_depth'],\n",
       "     'memory_size': 15926341}}},\n",
       "  'LightGBMClassifier_FULL_STACKER_l0': {'name': 'LightGBMClassifier_FULL_STACKER_l0',\n",
       "   'model_type': 'StackerEnsembleModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.17003107070922852,\n",
       "   'predict_time': None,\n",
       "   'val_score': None,\n",
       "   'hyperparameters': {'max_models': 25, 'max_models_per_type': 5},\n",
       "   'hyperparameters_fit': {},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 270549,\n",
       "   'bagged_info': {'child_type': 'LGBModel',\n",
       "    'num_child_models': 1,\n",
       "    'child_model_names': ['LightGBMClassifier_FULL'],\n",
       "    '_n_repeats': 1,\n",
       "    '_k_per_n_repeat': [1],\n",
       "    '_random_state': 0,\n",
       "    'low_memory': True,\n",
       "    'bagged_mode': False,\n",
       "    'max_memory_size': 539473,\n",
       "    'min_memory_size': 539473},\n",
       "   'stacker_info': {'num_base_models': 0,\n",
       "    'base_model_names': [],\n",
       "    'use_orig_features': True},\n",
       "   'children_info': {'LightGBMClassifier_FULL': {'name': 'LightGBMClassifier_FULL',\n",
       "     'model_type': 'LGBModel',\n",
       "     'problem_type': 'multiclass',\n",
       "     'eval_metric': 'accuracy',\n",
       "     'stopping_metric': 'accuracy',\n",
       "     'fit_time': 0.17003107070922852,\n",
       "     'predict_time': None,\n",
       "     'val_score': None,\n",
       "     'hyperparameters': {'num_boost_round': 26,\n",
       "      'num_threads': -1,\n",
       "      'objective': 'multiclass',\n",
       "      'num_classes': 3,\n",
       "      'verbose': -1,\n",
       "      'boosting_type': 'gbdt',\n",
       "      'two_round': True},\n",
       "     'hyperparameters_fit': {'num_boost_round': 0},\n",
       "     'hyperparameters_nondefault': [],\n",
       "     'memory_size': 268924}}},\n",
       "  'CatboostClassifier_FULL_STACKER_l1': {'name': 'CatboostClassifier_FULL_STACKER_l1',\n",
       "   'model_type': 'StackerEnsembleModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.41738009452819824,\n",
       "   'predict_time': None,\n",
       "   'val_score': None,\n",
       "   'hyperparameters': {'max_models': 25, 'max_models_per_type': 5},\n",
       "   'hyperparameters_fit': {},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 56055829,\n",
       "   'bagged_info': {'child_type': 'CatboostModel',\n",
       "    'num_child_models': 1,\n",
       "    'child_model_names': ['CatboostClassifier_FULL'],\n",
       "    '_n_repeats': 1,\n",
       "    '_k_per_n_repeat': [1],\n",
       "    '_random_state': 1,\n",
       "    'low_memory': True,\n",
       "    'bagged_mode': False,\n",
       "    'max_memory_size': 56113511,\n",
       "    'min_memory_size': 56113511},\n",
       "   'stacker_info': {'num_base_models': 10,\n",
       "    'base_model_names': ['RandomForestClassifierGini_STACKER_l0',\n",
       "     'RandomForestClassifierEntr_STACKER_l0',\n",
       "     'ExtraTreesClassifierGini_STACKER_l0',\n",
       "     'ExtraTreesClassifierEntr_STACKER_l0',\n",
       "     'LightGBMClassifier_STACKER_l0',\n",
       "     'RandomForestClassifierGini_FULL_STACKER_l0',\n",
       "     'RandomForestClassifierEntr_FULL_STACKER_l0',\n",
       "     'ExtraTreesClassifierGini_FULL_STACKER_l0',\n",
       "     'ExtraTreesClassifierEntr_FULL_STACKER_l0',\n",
       "     'LightGBMClassifier_FULL_STACKER_l0'],\n",
       "    'use_orig_features': True},\n",
       "   'children_info': {'CatboostClassifier_FULL': {'name': 'CatboostClassifier_FULL',\n",
       "     'model_type': 'CatboostModel',\n",
       "     'problem_type': 'multiclass',\n",
       "     'eval_metric': 'accuracy',\n",
       "     'stopping_metric': 'accuracy',\n",
       "     'fit_time': 0.41738009452819824,\n",
       "     'predict_time': None,\n",
       "     'val_score': None,\n",
       "     'hyperparameters': {'iterations': 22,\n",
       "      'learning_rate': 0.1,\n",
       "      'random_seed': 0,\n",
       "      'allow_writing_files': False,\n",
       "      'eval_metric': 'Accuracy'},\n",
       "     'hyperparameters_fit': {'iterations': 22},\n",
       "     'hyperparameters_nondefault': [],\n",
       "     'memory_size': 57682}}}}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor = task.load(model_folder) # unecessary here, just demonstrates how one would reload the predictor in practice\n",
    "predictor._learner.persist_trainer()\n",
    "predictor.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `persist_trainer()` function above loads all models required for prediction from disk into memory. Doing this deserialization in advance and only one time is crucial for efficient online inference. We're now ready to efficiently predict on one datapoint at a time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>weight</th>\n",
       "      <th>admission_type_id</th>\n",
       "      <th>discharge_disposition_id</th>\n",
       "      <th>admission_source_id</th>\n",
       "      <th>time_in_hospital</th>\n",
       "      <th>payer_code</th>\n",
       "      <th>medical_specialty</th>\n",
       "      <th>num_lab_procedures</th>\n",
       "      <th>...</th>\n",
       "      <th>examide</th>\n",
       "      <th>citoglipton</th>\n",
       "      <th>insulin</th>\n",
       "      <th>glyburide-metformin</th>\n",
       "      <th>glipizide-metformin</th>\n",
       "      <th>glimepiride-pioglitazone</th>\n",
       "      <th>metformin-rosiglitazone</th>\n",
       "      <th>metformin-pioglitazone</th>\n",
       "      <th>change</th>\n",
       "      <th>diabetesMed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Female</td>\n",
       "      <td>[70-80)</td>\n",
       "      <td>?</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"1\"</td>\n",
       "      <td>\"7\"</td>\n",
       "      <td>5.3</td>\n",
       "      <td>MC</td>\n",
       "      <td>?</td>\n",
       "      <td>37</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 46 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    gender      age weight admission_type_id discharge_disposition_id  \\\n",
       "29  Female  [70-80)      ?               \"1\"                      \"1\"   \n",
       "\n",
       "   admission_source_id  time_in_hospital payer_code medical_specialty  \\\n",
       "29                 \"7\"               5.3         MC                 ?   \n",
       "\n",
       "    num_lab_procedures  ...  examide  citoglipton  insulin  \\\n",
       "29                  37  ...       No           No       No   \n",
       "\n",
       "    glyburide-metformin  glipizide-metformin glimepiride-pioglitazone  \\\n",
       "29                   No                   No                       No   \n",
       "\n",
       "   metformin-rosiglitazone metformin-pioglitazone  change diabetesMed  \n",
       "29                      No                     No      No         Yes  \n",
       "\n",
       "[1 rows x 46 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: accuracy on test data: 0.43333333333333335\n",
      "Evaluations on test data:\n",
      "{\n",
      "    \"accuracy\": 0.43333333333333335,\n",
      "    \"accuracy_score\": 0.43333333333333335,\n",
      "    \"balanced_accuracy_score\": 0.28698752228163993,\n",
      "    \"matthews_corrcoef\": -0.10475656017578482\n",
      "}\n",
      "/Users/jonasmue/virtual/agmaster/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "Detailed (per-class) classification report:\n",
      "{\n",
      "    \"<30\": {\n",
      "        \"precision\": 0.0,\n",
      "        \"recall\": 0.0,\n",
      "        \"f1-score\": 0.0,\n",
      "        \"support\": 2\n",
      "    },\n",
      "    \">30\": {\n",
      "        \"precision\": 0.25,\n",
      "        \"recall\": 0.2727272727272727,\n",
      "        \"f1-score\": 0.2608695652173913,\n",
      "        \"support\": 11\n",
      "    },\n",
      "    \"NO\": {\n",
      "        \"precision\": 0.5555555555555556,\n",
      "        \"recall\": 0.5882352941176471,\n",
      "        \"f1-score\": 0.5714285714285715,\n",
      "        \"support\": 17\n",
      "    },\n",
      "    \"accuracy\": 0.43333333333333335,\n",
      "    \"macro avg\": {\n",
      "        \"precision\": 0.26851851851851855,\n",
      "        \"recall\": 0.28698752228163993,\n",
      "        \"f1-score\": 0.2774327122153209,\n",
      "        \"support\": 30\n",
      "    },\n",
      "    \"weighted avg\": {\n",
      "        \"precision\": 0.4064814814814815,\n",
      "        \"recall\": 0.43333333333333335,\n",
      "        \"f1-score\": 0.4194616977225673,\n",
      "        \"support\": 30\n",
      "    }\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions:  ['NO' 'NO' 'NO' '>30' 'NO' '>30' '>30' 'NO' 'NO' 'NO' 'NO' 'NO' '>30' 'NO'\n",
      " 'NO' 'NO' '>30' 'NO' '>30' '>30' 'NO' '>30' '>30' 'NO' '>30' 'NO' '>30'\n",
      " 'NO' 'NO' '>30']\n"
     ]
    }
   ],
   "source": [
    "num_test = 30\n",
    "preds = np.array(['']*num_test, dtype='object')\n",
    "\n",
    "for i in range(30):\n",
    "    datapoint = test_data_nolab.iloc[[i]]\n",
    "    pred_numpy = predictor.predict(datapoint)\n",
    "    preds[i] = pred_numpy[0]\n",
    "\n",
    "display(datapoint)\n",
    "perf = predictor.evaluate_predictions(y_test[:num_test], preds, auxiliary_metrics=True)\n",
    "print(\"Predictions: \", preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "[**AutoGluon Documentation** (autogluon.mxnet.io)](https://autogluon.mxnet.io/api/autogluon.task.html)\n",
    "\n",
    "[**Train/Deploy AutoGluon in the Cloud**](https://github.com/awslabs/autogluon/#traindeploy-autogluon-in-the-cloud)\n",
    "\n",
    "[**Build, Train, and Deploy a Machine Learning Model\n",
    "with Amazon SageMaker**](https://aws.amazon.com/getting-started/hands-on/build-train-deploy-machine-learning-model-sagemaker/)\n",
    "\n",
    "Kervizic, J. [**Overview of Different Approaches to Deploying Machine Learning Models in Production**](https://www.kdnuggets.com/2019/06/approaches-deploying-machine-learning-production.html). *KDNuggets*, 2019.\n",
    "\n",
    "Fakoor et al. [**Fast, Accurate, and Simple Models for Tabular Data\n",
    "via Augmented Distillation**](https://arxiv.org/abs/2006.14284). *Arxiv*, 2020.\n",
    "\n",
    "Bucila et al. [**Model Compression**](https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf). In: *KDD*, 2006. \n",
    "\n",
    "Hinton et al. [**Distilling the Knowledge in a Neural Network**](https://arxiv.org/abs/1503.02531). In: *NIPS Deep Learning Workshop*, 2014. \n",
    "\n",
    "Parr et al. [**Beware Default Random Forest Importances**](https://explained.ai/rf-importance/). *explained.ai*, 2018.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
